0.3.8

Recent Updates

• 0.3.8
  • engines/tax_pres.py
    • Used Cython to this file
    • Enhanced taxonomy presentation processing
    • Fixed calculation linkbase processing errors
    • Improved memory management for large filings
    • Optimized fact extraction for disclosures
    • Added better error handling for label links
    • Enhanced logging and memory usage tracking
  • engines/ins_facts.py
    • Moved fact_df = ins_facts(xid, tax) to
  • edgar/loader.py added get_xbrl_df() to replace get_fact_df()

0.3.8

OpenESEF 0.3.8 - tax_pres.py Update Notes

Bug Fixes

• Fixed error when picking dump and loading have different numpy version on different computers.
• Fixed error handling for calculation linkbase processing where Cython function objects were missing 'lower' and 'endswith' attributes
• Fixed string concatenation error when processing label links with NoneType values
• Improved memory management during fact extraction to prevent excessive memory usage

Enhancements

1. engines/tax_pres.py Improved Fact Processing

   • Added more efficient fact extraction for disclosures with minimal information capture
   • Reduced memory footprint by limiting stored information for disclosure facts
   • Added 'is_disclosure' flag to clearly distinguish between primary statements and disclosures

2. edgar/loader.py added get_xbrl_df() to replace get_fact_df()

0.3.7

Recent Updates

• 0.3.7
  • Taxonomy now processes calculation networks
  • Added engines.tax_pres.tax_calc_df() to get the calculation network dataframe

0.3.1

OpenESEF 0.3.1 Release Notes

Improvements

Memory Management

• Added robust memory monitoring and management system
• Implemented memory usage checks throughout XBRL processing pipeline
• Added configurable memory thresholds (default 16GB) to prevent OOM errors
• Improved cleanup of large objects after processing

XBRL Processing

• Enhanced fact extraction with better statement classification
• Improved handling of multi-statement concept appearances
• Added support for tracking concept order in financial statements
• Better handling of segment and dimension information

Performance

• Optimized taxonomy presentation processing
• Added force_reload option for fact DataFrame generation
• Reduced memory footprint during filing processing

Error Handling

• Better error reporting for memory-related issues
• Improved logging throughout the processing pipeline
• Added graceful handling of large integer values

Bug Fixes

• Resolved issues with concept tracking across multiple statements
• Fixed handling of large integer values in filing processing
• Improved cleanup of temporary objects

Breaking Changes

None

Usage Notes

Memory thresholds can be configured when calling processing functions

0.3.0

taxpres.py - Taxonomy Presentation Processing Module

This module provides functionality for processing XBRL taxonomy presentation linkbases and
extracting structured concept information. It helps organize concepts into statements and
validates segment/dimension information.

Key Classes:

TaxonomyPresentation
Main class that processes taxonomy presentation networks and organizes concepts into
primary statements and disclosures.

Key Functions:

get_presentation_networks(taxonomy)
Extracts presentation networks from a taxonomy by examining linkbases and base sets.

get_network_details(tax, network, reporter)
Processes a presentation network to extract concept details and relationships.

get_child_concepts(reporter, network, concept, taxonomy, visited=None)
Recursively extracts child concepts from a presentation network hierarchy.

process_children(reporter, network, parent, concepts, grandparent_qname)
Helper function to process child concepts in a presentation network.

ins_facts(xid, tax)
Extracts facts from an XBRL instance document and organizes them based on the
presentation structure.

Example Usage:

Create a TaxonomyPresentation instance

t_pres = TaxonomyPresentation(taxonomy, reporter)

Get facts from an instance document

fact_df = ins_facts(xbrl_instance, taxonomy)

Access statement information

print(t_pres.statement_concepts) # Concepts in primary statements
print(t_pres.disclosure_concepts) # Concepts in disclosures
print(t_pres.statement_dimensions) # Allowed dimensions per statement

Classes:

TaxonomyPresentation:
Attributes:
tax: The taxonomy object being processed
reporter: TaxonomyReporter instance for label handling
concept_df: DataFrame containing all concepts
allowed_segments_by_statement: Dict mapping statements to allowed segments
concept_dict: Dict containing all concepts
statement_concepts: Dict containing primary statement concepts
disclosure_concepts: Dict containing disclosure concepts
statement_dimensions: Dict containing allowed dimensions per statement
so_name: Name of Statement of Operations
fp_name: Name of Financial Position statement
cf_name: Name of Cash Flow statement

Methods:
    populate_concept_df(): Creates DataFrame from concept dictionaries
    _is_primary_statement(role_name): Determines if a role represents a primary statement
    _process_network_dimensions(network, statement_name): Processes dimensions in a network
    _validate_segment(segment_data, statement_name): Validates segment data against statement
    _process_taxonomy(): Main method to process taxonomy and build concept dictionaries
    is_valid_concept(concept_qname): Checks if a concept exists in presentation
    get_concept_info(concept_qname): Gets detailed information about a concept
    is_valid_segment(concept_qname, segment_data, statement_name): Validates segment data

Notes:

• The module assumes a standard XBRL taxonomy structure with presentation linkbases
• Primary statements are identified using keyword matching in role names
• Segment validation supports both axis/member and dimension/member terminology
• Period types and other attributes are obtained from the concept definitions

0.2.5

It can now print out an income statement exactly the same as reported. At least for Tesla 2019. https://github.com/reeyarn/openesef/blob/master/examples/ScreenshotTSLA.png

0.2.9

 fact_id                                              label                                       concept_name  value_mln        value  fact_included

0 F_000123 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 19952.0 19952000000 True
1 F_000126 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 869.0 869000000 True
2 F_000129 Sales Revenue Automotive SalesRevenueAutomotive 20821.0 20821000000 True
3 F_000132 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 1531.0 1531000000 True
4 F_000135 Sales Revenue Services And Other Net SalesRevenueServicesAndOtherNet 2226.0 2226000000 True
5 F_000138 Revenues Revenues 24578.0 24578000000 True
6 F_000141 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 15939.0 15939000000 True
7 F_000144 Cost Of Automotive Leasing CostOfAutomotiveLeasing 459.0 459000000 True
8 F_000147 Cost Of Revenues Automotive CostOfRevenuesAutomotive 16398.0 16398000000 True
9 F_000150 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 1341.0 1341000000 True
10 F_000153 Cost Of Services And Other CostOfServicesAndOther 2770.0 2770000000 True
11 F_000156 Cost Of Revenue CostOfRevenue 20509.0 20509000000 True
12 F_000159 Gross Profit GrossProfit 4069.0 4069000000 True
13 F_000162 Research And Development Expense ResearchAndDevelopmentExpense 1343.0 1343000000 True
14 F_000165 Selling General And Administrative Expense SellingGeneralAndAdministrativeExpense 2646.0 2646000000 True
15 F_000168 Restructuring And Other Expenses RestructuringAndOtherExpenses 149.0 149000000 True
16 F_000170 Operating Expenses OperatingExpenses 4138.0 4138000000 True
17 F_000173 Operating Income Loss OperatingIncomeLoss -69.0 -69000000 True
18 F_000176 Investment Income Interest InvestmentIncomeInterest 44.0 44000000 True
19 F_000179 Interest Expense InterestExpense 685.0 685000000 True
20 F_000182 Other Nonoperating Income Expense OtherNonoperatingIncomeExpense 45.0 45000000 True
21 F_000185 Income Loss From Continuing Operations Before ... IncomeLossFromContinuingOperationsBeforeIncome... -665.0 -665000000 True
22 F_000188 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 True
23 F_000194 Net Income Loss Attributable To Noncontrolling... NetIncomeLossAttributableToNoncontrollingInterest 87.0 87000000 True
24 F_000197 Net Income Loss NetIncomeLoss -862.0 -862000000 True
25 F_000200 Earnings Per Share Basic EarningsPerShareBasic NaN -4.92 True
26 F_000203 Earnings Per Share Diluted EarningsPerShareDiluted NaN -4.92 True
27 F_000206 Weighted Average Number Of Shares Outstanding ... WeightedAverageNumberOfSharesOutstandingBasic NaN 177000000 True
28 F_000209 Weighted Average Number Of Diluted Shares Outs... WeightedAverageNumberOfDilutedSharesOutstanding NaN 177000000 True
29 F_000645 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 594.0 594000000 False
30 F_000655 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 218.0 218000000 False
31 F_000664 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 186.0 186000000 False
32 F_000685 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 19212.0 19212000000 False
33 F_000687 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 146.0 146000000 False
34 F_000689 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 594.0 594000000 False
35 F_000691 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 1000.0 1000000000 False
36 F_000693 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 2226.0 2226000000 False
37 F_000695 Revenue From Contract With Customer Excluding ... RevenueFromContractWithCustomerExcludingAssess... 23178.0 23178000000 False
38 F_000697 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 869.0 869000000 False
39 F_000699 Operating Leases Income Statement Lease Revenue OperatingLeasesIncomeStatementLeaseRevenue 531.0 531000000 False
40 F_000701 Revenues Revenues 24578.0 24578000000 False
41 F_000733 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 27.0 27000000 False
42 F_001752 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 False
43 F_001764 Income Loss From Continuing Operations Before ... IncomeLossFromContinuingOperationsBeforeIncome... -665.0 -665000000 False
44 F_001785 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 False
45 F_001858 Income Tax Expense Benefit IncomeTaxExpenseBenefit 110.0 110000000 False
46 F_001891 Cost Of Goods And Services Sold CostOfGoodsAndServicesSold 350.0 350000000 False
47 F_001989 Revenues Revenues 23047.0 23047000000 False
48 F_001992 Gross Profit GrossProfit 3879.0 3879000000 False
49 F_001995 Revenues Revenues 1531.0 1531000000 False
50 F_001998 Gross Profit GrossProfit 190.0 190000000 False
51 F_002001 Revenues Revenues 12653.0 12653000000 False
52 F_002004 Revenues Revenues 2979.0 2979000000 False
53 F_002007 Revenues Revenues 1590.0 1590000000 False
54 F_002010 Revenues Revenues 1201.0 1201000000 False
55 F_002013 Revenues Revenues 6155.0 6155000000 False
56 F_002016 Revenues Revenues 24578.0 24578000000 False

0.2.0

• Integrated the code from farhadab/sec-edgar-financials;
• Using memfs to load XBRL files from inside EDGAR's full-text file without writing to tempdir

0.1.0 testing alpha release

testing