luminesce-examples (WIP)

This repo contains reference examples for Luminence.

This project is a WIP.

branch	status
master
master

Run using Docker

You can run with docker as follows.

Build the image:

docker build . -t luminesce-examples-runner

Run a container by passing auth credentials as environment variables:

docker run \
-e FBN_LUSID_API_URL=<API_URL> \
-e FBN_PASSWORD=<PASSWORD> \
-e FBN_APP_NAME=<APP_NAME> \
-e FBN_USERNAME=<USERNAME> \
-e FBN_CLIENT_SECRET=<CLIENT_SECRET> \
-e FBN_CLIENT_ID=<CLIENT_ID> \
-e FBN_TOKEN_URL=<TOKEN_URL> \
-e FBN_DRIVE_API_URL=<DRIVE_URL> \
-e FBN_LUMI_API_URL=<LUMI_URL> \
luminesce-examples-runner -start_dir=examples/drive

Running locally

You can run one or more of these examples locally by using the --secrets and --start_dir parameters.

For example, to run the Drive examples only:

python runner/run.py --secrets=secrets/secrets.json --start_dir=examples/drive

You can also choose to keep the sample Drive files created by the runner:

python runner/run.py --secrets=secrets/secrets.json --start_dir=examples/drive --keepfiles

List of examples

💡 The files in a directory are numbered if they need to be run in order 💡

Statistical functions

• Fuzzy search two files

Insights

• Count of requests per lusid method

Instruments

• Query instruments
• Set properties from csv
• Upload bond instruments
• Upload equity instruments
• Upload future instruments
• Upload fx forward
• Upload simple instruments
• Upload term deposit instruments

Complex bonds

• Step 01: Fixed schedule view
• Step 02: Step schedule view
• Step 03: Upload short front sinking complex bond insturments
• Step 1: Fixed schedule view
• Step 2: Step schedule view
• Step 3: Upload short front sinking complex bond insturments

Run valuation

• Step 1: Create instruments
• Step 2: Create portfolio
• Step 3: Upload quotes
• Step 4: Upload transactions
• Step 5: Run simple valuation
• Step 6: Upload external valuations
• Step 7: Run valuation with srs

Run a reconciliation

• Step 1: Create transaction portfolio
• Step 2: Create instruments
• Step 3: Create holdings
• Step 4: Create reconciliation view
• Step 5: Run recon with notifications

Holdings

• Step 1: Create instruments
• Step 2: Create and inline txn property
• Step 3: Create transaction portfolio
• Step 4: Adjust holdings
• Step 5: Cancel holdings
• Step 6: Set holdings
• Step 7: Call holdings

Abor

• Step 01: Create a portfolio
• Step 02: Create equity instruments
• Step 03: Create bond instruments
• Step 04: Create instrument properties
• Step 05: Assign properties to instrument
• Step 06: Create transactions
• Step 07: Upload instrument prices
• Step 08: Upload fx rates
• Step 09: Create cash transactions
• Step 10: Add transaction properties
• Step 10: Create a chart of accounts
• Step 11: Add accounts to chart of accounts
• Step 12: Create posting module
• Step 13: Create posting rules
• Step 14: Create an abor configuration
• Step 15: Create abor
• Step 16: Create journal entry lines
• Step 17: Create general ledger profile
• Step 18: Create general ledger profile mappings
• Step 19: Generate trial balance
• Step 20: Run a trial balance check

Properties

• Create instrument properties
• Create properties from csv

Corporate actions source

• Create corporate action source
• Read corporate action source

Portfolios

• Create transaction portfolio

Transactions

• Upload transactions from csv
• Upload transactions from excel
• Upload transactions from txt
• Upload transactions from xml

Reference portfolios

• Step 1: Upload instruments
• Step 2: Create reference portfolio
• Step 3: Upload constituents

Quotes

• Upload fx quotes

Relationships

• Step 1: Create properties
• Step 2: Upsert instrument properties
• Step 3: Create legal entities
• Step 4: Create portfolios
• Step 5: Assign lei to portfolio
• Step 6: Create relationship definition
• Step 7: Call back custodians

Returns

• Step 1: Create transaction portfolio
• Step 2: Upload portfolio returns
• Step 3: Calculate aggregate returns

Delete entities

• Step 1: Delete quotes
• Step 2: Delete instruments
• Step 3: Delete portfolios
• Step 4: Delete properties
• Step 5: Delete coa
• Step 6: Delete gl profiles

Run a recon holdings in different scopes

• Step 1: Create transaction portfolios in two scopes
• Step 2: Create instruments
• Step 3: Upload abor transactions
• Step 4: Upload ibor holdings
• Step 5: Create reconciliation view
• Step 6: Run recon workflow
• Step 7: Run recon with generic reconciliation provider

Drive

• Create and move file in drive
• Create log file on error
• Read a file from drive
• Read an excel file from drive
• Save data into drive
• Unformatted string to table

Pdf generation

• Step 1: Load data from drive
• Step 2: Generate pdf

Horizon

• Query company data from gleif

System

• Append inline properties to system configuration
• Error handling details to file
• Iif and case when statements
• Load one cell of data to table by delimiters
• Pivot data
• Random prices generator
• To utc and iso
• Using datetimes

For loops with cross apply

• Step 1: Create instrument upsert view
• Step 2: Loop over csv file

View management

• Step 1: Create view with no params
• Step 2: Fetch sql used to create view
• Step 3: Current content of view
• Step 4: Show view history
• Step 5: Show view dependencies
• Step 6: Show dependencies on view
• Step 7: Delete a view
• Step 8: Run table of query strings

Basic data integrity

• Basic data integrity

Check for duplicates

• Step 0: Load instruments into lusid
• Step 1: Create duplicate check view

Check for price outliers

• Step 0: Create instrument property definitions
• Step 1: Setup instruments with properties
• Step 2: Upload quotes
• Step 3: Create iqr checker view
• Step 4: Create price outlier view
• Step 5: Run price outlier view

Reconcile instruments

• Step 1: Create instrument properties
• Step 2: Setup instrument with properties
• Step 3: Load instruments into source a
• Step 4: Load instruments into source b
• Step 5: Run reconciliation

Cross sectional outliers

• Step 0: Create properties
• Step 1: Upload instrumets
• Step 2: Create cross sectional outlier view
• Step 3: Run cross sectional outlier view

Check for missing instrument fields

• Step 1: Create instrument properties
• Step 2: Load instruments into lusid
• Step 3: Check for missing instrument fields

File orchestration

• Step 1: Files for testing
• Step 2: Create error file
• Step 3: Create logger view
• Step 4: Create file orchestration view
• Step 5: Run file load

Automated testing

We run automated tests on the SQL files in this project via GitHub Actions. The configurtion for these tests live in the .github/workflows directory.

Many of our tests require setup data. To create this data, there is a process where the testing runner will search for a _data directory wherever it finds .sql files. Then, two things happen:

1. If there are data files in the _data directory, the runner will upload these to a luminesce-examples folder in LUSID Drive
2. If there is a setup.py file in this directory, the runner will run the setup.py file. We use this setup.py file to configure recipes and other configurations we don't want to setup via Luminesce.

Sample structure below:

upload_equity_instruments.sql
upload_bond_instruments.sql
_data
    eq_instruments.csv
    instruments.txt
    setup.py

⚠️ This file is generated, any direct edits will be lost. For instructions on how to generate the file, see docgen.