Data

I like the word canonicalization, which made an important appearance at the 2026 ICM! Here I describe the final cleaned dataset that will be canonicalized for the rest of the project. The three sets are price store, corporate-actions ledger and universe definitions. The time period is 2016-01-04 to 2026-07-31, and the data is sourced from Alpaca's SIP feed, various online sources for events and information (e.g. public GitHub stores for point-in-time membership) and public filings to reconcile events with prices.

Prices

One parquet file per security and per adjustment basis (raw or adjusted), saved as data/adjusted/<SERIES>/1d.parquet and data/unadjusted/<SERIES>/1d.parquet. There are 724 series in total, including 12 benchmarks, with 1,698,349 total daily OHLCV bars across the whole universe. The two price series for each security share identical row counts and dates.

The adjusted series is total-return, with dividends, splits, events applied to rebase the series for total shareholder value. Each file also carries metadata inside the parquet footer, including the ticker, timeframe, adjustment mode, feed, as-of date and the window requested. See note on parquet storage.

When we load the parquet file into memory, which can be simply done with pandas read_parquet function without any extra parameters as follows: df = pd.read_parquet("data/adjusted/AAPL/1d.parquet") where df will be a Pandas dataframe with size (2659, 5). The columns are open, high, low, close as float64, and volume as int64. The index is a proper DatetimeIndex like usual.

Another thing to note is that the index is timezone-aware UTC. For daily bars, we don't need to handle this; however, events arrive with naive dates, but we can simply drop the timezone with index.tz_localize(None).normalize() on one side.

For adjustment, the series is rebased backwards from today, i.e. the adj/raw ratio is 1.0 at the latest date. I want to cover here how adjustments work with an example, for my own understanding and also completeness. We imagine that, say, AAPL starts from $100, pays out a 2% dividend on a date, and then goes through a 2:1 split, ending at $51; we assume no other true movements on event dates for simplicity, but these are tracked normally since the ex-price and split are reflected at open. Here is how this would work over 6 dates:

Note that the adjusted series is only computed ex-post, after the entire raw-price series and events are reconciled. After applying the adjustment, we then look at the time-series again and verify that the adj return is the total return that a shareholder receives. The event at t produces an adjusted factor for the entire trailing part of the series (we basically shift the entire series to the appropriate "artificial" level that events cause, while preserving the actual underlying returns from the economic movement e.g. the adjusted series is blind to things like splits and dividends, constant through them up to true movements). This means that the adjusted factor at d1 for example contains all future events (in this case the dividend and the split, and there is no systematic way to determine what such a factor is made up of without information about the corporate actions).

Corporate actions

The corporate actions are organized into two tables. The first table is an event table, and documents what happened to which security and what dates; it has 20,460 rows e.g.

The outcomes table has 20,522 rows and contains what a shareholder receives. The distinction is there because a single event can sometimes have multiple things delivered to an investor. The second table has rows like

The true schematic representation is more complicated with additional fields. I included the CELG stock and cash merger event because it leads us to some decisions as discussed earlier in the segue entry. Celgene (CELG) was acquired by Bristol-Myers Squibb, completing on 2019-11-20. The three outcomes for a holder were $50.00 cash credited, 1 BMY share, which we also have prices for in our prices dataset, and finally, 1 BMY.RT, which is a Contingent Value Right: Celgene shareholders received a tradable CVR entitling them to $9 in cash if three drugs were approved by specified deadlines. If any were missed, the CVR was worth 0 and expired. The approvals did not happen on time and so the right terminated at zero! Handling these events is a lot of effort since we don't know a priori how to price these sorts of assets, and the strategy also should not be impacted by optimizing these kinds of choices; the decision we make is that we ignore the securities that have these special events by not holding them during these windows; this does not introduce any lookahead since these events are pre-announced!

I also want to list the total types of events present in our dataset, and the respective counts. As you can see, most of these are dividends and splits, which is expected.

86 of the price series have no corporate actions, e.g. recent listings (like PLTR, HOOD) and companies with full history but just no dividend or split events at all (like ADBE, AMD).

All events carry the effective date (e.g. for mergers it's when a security stops existing) and for dividends this is the ex-date. They also have CUSIP (from Alpaca), the relevant price series, and a source URL when we had to resort to online search or filing. We also mark outcomes if they lie outside the universe (e.g. receive an asset that we don't have prices for). We also mark contingent outcomes and default election options when appropriate.

Universe

The point-in-time S&P 500 membership is present in cohort_price_identities.csv. It has 11,087 rows across 22 semiannual cohorts running from 2016-H1 to 2026-H2, each mapping a membership ticker to one price series, and also the time period when the map is valid. Of course, this is not an injective mapping, since the distinct price series are 712 (ex-benchmarks). Many distinct price series are in multiple cohorts, as is natural (e.g. META is in all of them). This is part of the cleaned dataset in the sense that we have the correct mappings now. For the backtest, the symbols are completely arbitrary; all we need is to assign to each price series a cohort list where it is eligible and that is it. We also recorded ticker aliases, identity transitions, zero-volume exclusions, gaps etc.

We now have to make some decisions about how we plan to handle some of the data considerations; a lot of them are either directly backtesting decisions or primarily concern backtesting and not the data per se, but I will still treat this as part of the cleaning process, since our decisions will affect what data continues on from this point on. In the next entry, I will produce the attributes of the final data that will proceed into the pairs formation, backtest and execution layers of this project, which at this point are still wide open.

At this point, we have to decide how to handle all the non-routine corporate events in our dataset. My initial instinct is to not micromanage the corporate actions optimization in the backtest, and simply prune these events, even though this introduces look-ahead. I counted all the sorts of events that in an ideal world I would not want to write if-conditions for in my backtest: things like elections, contractual rights without a clear value, spin-offs where we actually don't have price series yet. Of 724 price series, 121 have terminal events and 37 receive something while they are eligible to trade. Most of the data (20,131 events) are routine dividends and splits, and we have already decided how to handle these.

The 37 symbols are like MMM, GE, PFE, T, HON, RTX, DHR, which I definitely want to retain; the nice thing about many of these corporate actions are that they are pre-announced, meaning we can still ignore these specific time-periods without having to worry about pruning the data, and without introducing look-ahead, meaning do not let the strategy form pairs where the holding period would include those dates; this could be forced from the start, that is, exit any position before the ex-date of an announced non-routine corporate event and do not form pairs involving these events. It seems to me that a cointegration based strategy should do this anyway, since our confidence in a pair relationship probably degrades with these types of events anyway (e.g. an entire subset of the business leaves the company). The other things we need to make decisions for are as follows:

The events that the exit rule excludes are checked to ensure that they were pre-announced using the Alpaca News API and the announcement endpoint and historical news data, or primary sources e.g. Newfield/Encana. All 168 announced non-routine events now have a known date, an exit date and an effective date, where the exit date is the first open after the close of the day by which we know the announcement, meaning a flag for us to close the position if it is open, potentially. How the pairs relations react to these events is maybe interesting and something worth exploring. This allows us to ignore signals during these periods for the respective securities while also claiming no lookahead. The remaining market exit events are preserved and exist as genuine and unknowable events before they occur.

Note that since sector labels also constantly change, and even the classification go through restructuring, I think using 2026 sector classification would be moderate to extreme lookahead in this study (giving us the right pairings in 2016 that may not have been predictable, which is the object we are trying to learn). Therefore, we might have to build point-in-time labellings as well if we want to do pairings modulo sector groupings, which I think we should try to do; the sector pairings are extremely natural consideration for pairs, and this information might turn out to be essential to avoid spurious or priced in relationships.

We book all the transit cash flow as soon as the effective date, meaning we don't wait for the payable date to receive the cash proceeds from a dividend for example. We basically assume no cash-constraints in our backtest other than just the initial gross capital deployed; we could also assume a larger initial investment and limit ourselves to a smaller portion for the L/S book (which would reduce the day-to-day returns since the denominator is higher) but this is an additional complication I do not yet see important enough to model; we also book all the negative cash flow at the effective dates, and we expect the exposure from this to be controlled, meaning we resort to measuring and reporting these effects (e.g. if the account ever runs into a cash-constrained situation). In downstream analysis of the strategy, we would also measure the margin account maintenance constraints and if they are ever violated. The second point I want to make is on the timing of the remaining terminal security situations which we may run into and the rule there; we book the dead security at zero for the account, and the unaffected leg of the L/S is then closed at the next day open (since the other option may introduce lookahead).

Note that in many cases, there is a nice symmetry with the long/short: we hope at least that such considerations "cancel" out e.g. the dividend yield, or takeover premiums, etc. Nevertheless, we intend to report these numbers for the strategy!

The securities that entered the index late did not have the dividend datasets before the windows where the security actually is traded (by our strategy); the reason is that we form pairs on adjusted series and use events and raw prices for execution. However, given that the data can be missing or incomplete or wrong outright, I thought it made sense to get the earlier corporate events for these stocks anyway, just to verify the adjusted series we use for signal generation. This meant that we added more events to the corporate actions dataset.

We also found more defects in the data by trying to reconcile prices and corporate events identity by identity (unfortunately this is what I had to resort to by the end of this gruelling process). One of these was that Alpaca gave corporate events that I could not find any evidence for otherwise.

The ticker reuse is basically the same issue as before; 52 adjusted price series needed more fixing.

I want to say that at this point I did not want to resort to independent correctness checks on the raw price histories (or even adjusted) from a secondary source (like Yahoo Finance); but I might do this in the future just for peace of mind; I am sure it would possibly reveal some inconsistencies. For now, I do think the data is "good enough" meaning any mistakes will be minor. Even so, the actual purpose of the project is for me to learn pairs trading strategies and test some of my own ideas; the whole reason I went on this 3 week data cleaning excursion is because I thought A) it is also valuable to learn and know the issues that arise in practice, and how complicated and hard backtesting can be (e.g. the number of pitfalls I have already encountered seem uncountable), and B) it seemed too academic to have a pairs trading project that doesn't even have some realistic-ish (something other than a turnover cost model) execution modelling since the issues of execution play directly into pairs. This is the first project where I will do truly point-in-time backtesting with a complete information block beyond day t, form signals, and simulate an actual portfolio. So for this purpose, I think the data at least now is largely historically accurate. Any mistakes that are too detrimental downstream (e.g. a signal "too good to be true") can be identified and fixed as the project proceeds.

While listing out the dividend counts for the securities in our universe (to check how many were dividend stocks at all), I discovered another issue with dividend data. BNY had 16 payments a year, every year. BNY Mellon is a quarterly payer, meaning we should have had 4 payments (12 additional payments).

Clearly the 12 extra payments leads us to the cause of the issue already. The issue was again ticker reuse (really tired of this now); BNY Mellon traded as BK until recently, and the ticker BNY belonged to a BlackRock municipal bond fund that paid monthly (thus the 12 additional dividends). Both payment schedules are ascribed to BNY.

This contamination had also been applied to the adjusted prices. My fault here is that when querying, I just used the symbol, when I should have differentiated based on CUSIP (CUSIP is an alphanumeric ID for securities, much more stable than tickers, and changes mostly on legal/structure changing events). However, the returned adjusted prices for BNY (the actual relevant price series) had a serious error when I was able to finally identify the true corporate events and the respective identities properly; Alpaca had computed the adjustment factors to be applied to BNY Mellon's raw price series using the dividend events, and the raw prices associated to the BlackRock fund! Meaning the adjustment factors you get from normalizing 3 cent payments to a $11 fund (BlackRock) were applied to a $45 stock (BNY Mellon); this is of course, an egregious oversight, especially for signal generation, especially compounded over 10 years. At this point, I also began wondering if we should use the raw prices and the corporate actions dataset we built (Alpaca + manual work) to build our own return series!

Alpaca also only populated 62% of the events with CUSIP, but for these, we were able to match identities and events properly. Other affected securities had to again be checked and verified manually. Here is the table of the surgery:

193 events and 179 adjustment factors were removed in total. META and TFC prices turned out to be clean, just the events we had collected for them were inflated. TFC is interesting: one of its own dividends fell on the date of a foreign one, meaning we had to separate the factor applied here in size!

Anyway, what this discovery means is that we need to run more checks on the actual data provided itself and the correctness, meaning we should check more comprehensively the events associated to a raw price series for an identity (not a symbol) and reconcile the adjusted prices by-hand given the corporate actions dataset, and not the other way around like we have been doing, i.e. the Alpaca provided data cannot be trusted fully.

I want to discuss the payment schedules associated with a dividend as the supplement to some of the fixes made in the full audit. The main dates associated to a dividend are the ex-date, the record date, and the payable date (also the declaration made by the board but this is what announces the dividend and the schedule in the first place). The record-date is the date that decides entitlement, i.e. the owner of the stock on the close on record-date is entitled to the dividend. However, since the trade also takes time to settle, there is also an ex-date, the first date that the stock trades without the right to the dividend (this is the date that the price reflects the loss of the dividend since buying it no longer entitles you to the dividend and the value is gone). The payable date is the date after the clearing process is finished and settlement happens, i.e. cash is received. This is usually 2-4 weeks later.

The important separation is between the record-date and the ex-date. Since 28 May 2024, a trade on day D settles on D+1 (we can call this T+1), because of improvements in speed and efficiency of the clearing/settlement process. This means that buying on the record date R is already too late, and you must buy on R-1 to be entitled; this means that the ex-date is now the record-date! The T+1 rule used to be T+2 and T+3 and so on.

There are also scattered gaps of 2, 3 and 4 because of weekends. The main exception to the rule is the following; if a distribution is worth roughly 25% or more of the share price, the exchange defers the ex-date to a business day after the payable date, meaning the dividend value stays attached to the share (as due bills) until it is actually paid out. This is to prevent a disruption such a large move in the price might cause. In our data, an example is AIV's dividend on 2025-10-16. Also, five dividends on 2025-01-10 have a record date of 2025-01-09, which was an unscheduled market closure, so the ex-date was the next day.

In our data, all cash dividends have the ex-date and effective-dates populated, but the record-date and the payable-date are only about 60% populated. There are some concerns associated with this - the data collection and cleaning were handled in the last entry (e.g. Alpaca filters using payable, so calling the actual eligible period removes additional dividends since the entitlement happens in the relevant period but the payment does not, and post-terminal dividends). A backtest concern is how to book the cash (ex-date determines the cash flow, but the cash only becomes available for financing on payable-date) etc. We will deal with this once we design the strategy!

We now ran final end-to-end checks on every issue we have observed yet and other possible ones I thought of on the entire dataset, and also reconciled the three datasets against each other (i.e. raw prices, adjusted prices, and corporate actions). The primary test was again the raw/adjusted comparison.

The first run found the $103.75 special dividend Dr Pepper Snapple paid in the Keurig transaction, and we recovered this figure with the prices exactly before reconciling the actual action. The issue here was the missing dividends attached to name changes or relabels (especially involving actual breaks/divergences in price histories) we mentioned earlier and this was fixed across the dataset. We also had to tune the ratio test to threshold for dividend moves, not just larger spin-offs or splits, while still not flagging penny-rounding noises.

Another rename was detected in the full audit (KORS and CPRI), which did not show up initially because they are listed at the same time i.e. they coexist (from 2016-H2 to 2018-H2), even though Michael Kors became Capri Holdings in 2018. Other issues identified here are listed in the table below.

Also note that there are 222 "outcomes" e.g. a stock dividend or shares distributed that are linked to other price series in our universe (meaning we can ascribe the value somewhere within the backtest), but there are also 77 events which live outside, e.g. we get value attached to a security outside the universe, with no price value attached to it; this presents another backtest issue: what to do with these assets? I think the simple and naive approach would be to get the raw price for them on the day we receive them, and just simply liquidate them, based on the raw price, with the execution friction and trading costs associated with them; any simplifying assumptions made for these (i.e. NMS security, no short-fee or the same level as the rest of the universe) should not hurt too much, and I think may be safely made given our scope. There are also 3 events that are a contractual right, CVR-type event (for example, Johnson & Johnson's acquisition of Abiomed in December 2022, where each ABMD share received $380 in cash plus one non-tradeable contingent value right worth up to $35 if specified milestones were met) that we also have to deal with.

This entry was renamed to consistency checks since that is what they are; we later discovered correctness issues with Alpaca adjusted series data at the source. See entries above. For the elections (meaning multiple choices given to shareholders), my initial standing is that our backtester should choose the no-vote outcome, meaning if a shareholder doesn't take any actions; this seems to be the standard choice since our study is not about execution and fundamental efficiency, but I will discuss this later when we close all the pending decisions about how to backtest and handle weird events.

We now collected and itemized the entire corporate actions dataset for the universe over the horizon. Early issues that used parts of this dataset were query-specific requests, and here we finally compile everything as the canonical corporate actions part of this dataset. The baseline came from Alpaca's corporate actions announcement endpoint, queried per symbol. When a record turned out to be missing (e.g. an event identified using the adj/raw series) or incomplete, we supplemented it with a primary source and a specific filing (e.g. EDGAR). 84 events carried such a supplement, and most of these are cited by an SEC filing. Three exchange-ending bank failures cite the FDIC notice, since there is no closing 8-K for example when a bank is taken into receivership.

The data is stored as two tables instead of forcing the entire thing into a pre-determined structure since corporate events can differ so wildly in their outcomes and the associated natural formatting. The first table is an event record i.e. it says what happened when and to which identity. The second table is the one that explains what the shareholder receives (a shareholder for the existing price-series for that identity): cash, replacement shares, distributions, or a contract e.g. a stock-and-cash merger has two outcomes, a spin-off has an outcome for both the parent and child, and an election deal has several option groups. A market exit has no outcomes in the pre-determined sense, especially for our design (since we still have to decide what happens in this case, and I relegated this to a downstream question about how to handle termination in our backtest/strategy design).

We obtained 19,013 events and 19,069 "outcomes", drawn from 20,873 Alpaca records. Most of these are just ordinary dividends, which is expected. Here is the table detailing the composition:

We checked coverage for the eligibility windows (rather than the whole period) since this is where it will matter for execution - we may later extend the dataset to cover the whole period for completeness, if needed here or for another project with different goals. All 89 histories that end inside a cohort window were labelled with an explanation for the termination (to help us figure out how the backtester ought to handle this, since this information would become available to a point-in-time trader, sometimes prior to the event). The 5 membership rows that did not have any price bars at all were also accounted for. The renaming issues that resulted in missed data were also reconciled here.

A note on proration and elections: investors are sometimes asked to choose between different packages, and the distribution can be prorated. E.g. carrying on the REIT example from before, MAC's April 2020 dividend declared $0.50 per share, but capped the cash at 20% (i.e. $0.10). The shareholders were asked to elect between a) cash: entitlement to $0.50, capped, b) stock (the equivalent value in shares) or c) no election, meaning they received 20% cash and the remaining in shares. This meant that any people preferring stock would cause the total cash pool available to grow larger for the distribution. I classified the handling of elections and dividend schedules (e.g. record date, payable date etc.) as downstream decisions (see above).

EDGAR is the Electronic Data Gathering, Analysis and Retrieval, the SEC's public database. It is the primary source we use when finding an online press release is not immediate or has friction, and also the authoritative source for the internal searches that I executed with automated calls, because every US-listed company must file its disclosures here and it is free with no login. Coverage runs from roughly 1993, and full-text search works from 2001 onward.

For corporate actions, I used the search feature for standard phrasing of the corporate event I was looking for, then filtered to form 8-K and a date window around the event (see below). The primary thing we usually look for is under the exhibits attached to the form, specifically EX-99.1, which is usually the press release. Here is the table for the form types; these are the names I use in the source links in many places in the log entries.

As stated in the adjusted seams entry, the issue revealed by comparing the raw series to the adjusted series may be broader than just the set of symbols that undergo erratic name changes. The sweep needed to be larger, and every single ticker may be susceptible to weird corporate actions behavior. The raw/adjusted ratio is in my opinion a very good discovery test to identify special events.

This is a good place to discuss how adjusted series are computed, specifically the back-adjustment procedure. The adjusted series tracks the position of a shareholder rather than the value of a single share itself, so the direction of the adjustment factor is governed by what happens to the shares per holder. Dividends, forward splits and spin-offs either add shares or take value out of the share itself, perhaps simultaneously, e.g. a share dividend; this can also be more complicated, e.g. a spin-off causing a share distribution and a simultaneous cash distribution, or a special dividend. This means that the historical price series is scaled down to accommodate the value-reducing event, meaning we end up pretending the entire company never had the value associated with this loss to give away, so the ratio of adjusted to raw steps up as you walk forward in time (as the raw is truly down but the adjusted is not, since it has by definition absorbed the loss a priori in its construction).

A step down requires the opposite to be true, i.e. that the shares per holder fall while the economic value is the same modulo true value changes. Reverse splits are the largest type of singular event that causes this (I am not sure about every single type of corporate action and their causes, since I have not conducted any reasonably comprehensive research about all types of actions). Other events within our dataset that also caused this were mergers or reorganizations with an exchange ratio below 1, and spin-offs accompanied by a consolidation. Johnson Controls combined with Tyco and each old share converted to 0.8357 new shares plus $5.7293 in cash, so a holder ends up with fewer shares than they started with and each remaining share has to stand for more of the position. Honeywell spun off its Aerospace business and ran a 2-for-1 reverse split at the same time, so each two old HON shares became one HON share plus one HONA share. It appears to me that these examples are countable and not too unwieldy to handle manually.

Testing all 724 series returned 23 material (meaning modulo penny-rounding noise) downward steps. 11 were reverse splits already handled correctly. Of the 12 remaining, 4 were also legitimate: two for AIV and two for SLG, each a reverse consolidation paired with a dividend, confirmed against the companies' own filings and thus retained. These were small enough that the initial screen on the size of the raw return that day could not tell them from defects. That left 8 genuine defects.

In six of the eight defects, the implied factor is exactly a share distribution ratio recorded for the event: 1/8 for CNX, 1/6 for SW, 1/3 for TGNA, 1/2 for RTX (the Otis leg of two simultaneous separations), 0.80 for EQT, and 0.8824 for IR. Unfortunately, it seems that Alpaca took the distribution ratio and applied it as though it were a share-consolidation factor, which is wrong in size and in direction, since a spin-off should scale history down and a consolidation scales it up. A consolidation factor rescales the entire history behind it, so CNX's adjusted prices were at exactly eight times raw for every session before the event. Every price level and every return spanning that date was wrong. The remaining two, HON and JCI, are genuine consolidations whose factors were the wrong size rather than the wrong kind.

The other side of the TT issue from before (the adjusted seam fix was incomplete) was also fixed here; this is the IR fix. What happened was that before the deal, the two companies being traded were Ingersoll-Rand plc, ticker IR, a diversified industrial holding two businesses, HVAC (Trane) and Industrial (compressors, tools), and Gardner Denver Holdings, ticker GDI, a completely different public company, also making industrial compressors. Ingersoll-Rand plc spun off its Industrial segment, and that segment immediately merged into Gardner Denver. Gardner Denver then became "Ingersoll Rand Inc." and took the ticker IR, while old Ingersoll-Rand plc, now just the HVAC business, became "Trane Technologies" and took the ticker TT. The ticker TT in our data (and the previous fix) refers to the continued business that became TT. Note the consequences to either shareholder, since this determines what should happen to the adjusted series: the old Ingersoll-Rand holder (what our data calls TT) has their share and has 0.8824 of IR (the new merged entity) shares distributed to them; the Gardner Denver holder just continues to have their share of the company (just a name change occurs, purely administrative), and any economic gain or loss is reflected in the value of the share they hold (the price post-deal), which is just dilution since the company issues new shares to new holders, but also gains an industrial business. For IR, there is no adjustment. Alpaca applied this adjustment to IR anyway (probably because of the ticker collision), causing a phantom 11.7% loss.

JCI is not a spin-off. Johnson Controls combined with Tyco, and each old JCI share converted to 0.8357 new shares plus $5.7293 in cash. Mixed consideration needs both terms, and the series had been rebuilt from only the share term. Three more defects were found by the adj/raw comparison where the adjustment size was wrong: AIV's $8.20 special dividend, the AIV to AIRC separation, and MAC's $0.50 dividend paid 20% cash and 80% stock.

Here is the story for AIV. I wanted to write it out since it needed multiple fixes and appears 4 times in the cleaning logs. Aimco is a REIT (a Real Estate Investment Trust is a company that owns, operates, or finances income-producing real estate), and a REIT has to distribute most of its taxable income to keep its tax treatment (at least 90%, and mostly 100%). Through 2020 a lot of them conserved cash by paying the bulk of a dividend in shares instead, and paired them with reverse stock splits to keep the share count steadier. AIV did this in February 2019 at 1-for-1.03119 and again in late 2020 at 1-for-1.23821. These downward steps were verified and retained. Same story with MAC. The two real AIV defects are two separate events: an $8.20 special dividend with an ex-date of 2020-11-03, and the separation on 2020-12-15 where holders kept their AIV share and received one share of Apartment Income REIT. The raw AIV close went 40.34 to 5.04 across that seam, so nearly all of the value moved into AIRC and what stayed behind under the AIV ticker is the smaller remainder.

Another issue that we accidentally discovered was that of Alpaca giving flat bars (constant price series with volume traded being 0, crucially) after a security stops trading, for various reasons. The carried quote is harder to identify for obvious reasons, unlike a gap, and it would have produced spurious pairs formations while being completely untradable. 891 of these were found and removed from both adjustment modes to produce true gaps instead. This is fine for look-ahead since the cessation of trading is actually observed in time, and the trading may or may not resume.

The first example that I discovered was SBNY: it had been holding a flat 70.00 for 509 sessions after Signature Bank was seized, a period over which the recovery to common was zero. Common shareholders are the residual claimant, paid only after depositors, secured and unsecured creditors and preferred stock are made whole, so in a receivership like this one they receive nothing and the quoted 70.00 was a price for a claim already worth zero. Its observed history now ends on 2023-03-10. We also removed the post-bankruptcy tails on DO, ESV and MNK, whose old equity had been cancelled. SMCI halted trading from 2018-08-22 to 2020-01-14, a gap of 510 days, but eventually resumed trading, which means it is a genuine gap.

Another issue that needed even more fixing was ticker reuse fusing two unrelated securities, e.g. with SNDK with the wrong as-of period passed from before. LB ran L Brands straight into LandBridge, which took the symbol years later. LB now follows L Brands into BBWI including the one-third VSCO distribution, and stops there. Historical DOW and historical DuPont are now separate identities that both stop at the 2017 DowDuPont merger boundary, with their 1.0 and 1.282 conversions deferred to when we get the corporate-action data. Old CHK is separated from the post-bankruptcy Chesapeake now trading as EXE. We now store a multi-key as the general fix, e.g. CHK_2020 and SNDK_2016, along with the tickers.

Again, I compared the adjusted/raw returns and found eight dates where the two differ by more than a known corporate action. Seven were real and were repaired in the full sweep in the above entry. The eighth is CHK_2020 on 2020-04-15, where the raw close goes 0.1312 to 16.38 and the implied factor is exactly 1/200. That is Chesapeake's genuine 200-for-1 reverse split. Its adjusted return that day is -37.6%.

The next issue has to do with coverage. 5 stocks exist in the universal set but have no bars at all, and 83 cohort rows sit below 90% coverage of their eligibility window, those 5 included (86 once the tradability pass in the entry above removed carried quotes). The issue here was the periodic semi-annual sampling on particular dates every year.

The five empty rows have their last traded price even before the cohort horizon starts. In each case the cohort was frozen at the 30 June or 31 December cutoff and a deal closed on the next trading day. The company was an index member at the cutoff and had ceased to exist by the time the cohort begins. The following list is the brief details and time stamps for this. I think the appropriate way to deal with these would just be to drop them from the “tradeable” set of securities, the ones actually used for the signal, executions and positions.

• CPGX, 2016-H2: last price 2016-06-30, eligibility starts 2016-07-01. Columbia Pipeline into TransCanada, effective 12:01am on 1 July at $25.50 per share in cash. Source
• GAS, 2016-H2: last price 2016-06-30, eligibility starts 2016-07-01. AGL Resources into Southern Company at $66 per share in cash; the stock ceased trading immediately before the open on 1 July. Source
• TE, 2016-H2: last price 2016-06-30, eligibility starts 2016-07-01. TECO Energy into Emera, completed 1 July. Source
• SCG, 2019-H1: last price 2018-12-31, eligibility starts 2019-01-01. SCANA into Dominion Energy, each SCANA share converting into 0.6690 Dominion shares. Source
• LLL, 2019-H2: last price 2019-06-28, eligibility starts 2019-07-01. L3 Technologies into L3Harris, each L3 share converting into 1.3 L3Harris shares. Source

The 83 low coverage are the 5 above plus 78 others (now 81); each of them ends within their respective cohorts because they stopped trading, mostly through acquisition or going private. Some of them stopped due to merger or outright failure (e.g. SIVB, SBNY and FRC). Examples include BRCM, PCP, GMCR, PCL, CAM, TWC, EMC, HOT, STJ, LLTC, HAR, RAI, LVLT and SPLS, but the detailed table is too long to list here; note this is the behaviour we do not know in advance, and precisely the behaviour we want to keep in the tradeable universe (since survivorship bias hurts us in these situations the most, e.g. going long one of these stocks that fail). Coverage runs from 0.8% (STJ, which lasted two sessions into 2017-H1) up to 89.8%. We will need a terminal-event rule in the backtester, and most likely some sort of explicit risk management for these situations; what comes to my head currently is a stop-loss rule, but it is too soon to address this.

The name relabeling issues from earlier produced another decision that has to be made: reconciling adjusted series (not just the correct price series labels) across the various seams that are introduced by the renaming debacles, e.g. two adjusted series across the rename and a spin-off. This also means that Alpaca adjustment does not account for every type of corporate action, and many must be resolved manually. The split series are always internally consistent, but due to rebasing issues, they cannot just be concatenated across the seam. See table below.

We use the corporate actions events to figure out the scales (note that Alpaca raw data can mess up here as well, e.g. duplicate dividends; a lot of the work has to be manual) and we rescale the earlier segment to match.

Some of the identified breaks were benign, e.g. a simple name change with no distribution to the continuing holder, or Alpaca correctly handled the adjustment. The changes were all reconciled by again taking the ratio of the adjusted and the raw prices. The first issue was with WYND, where the identified historical corporate event documented earlier is clearly not applied, and produces a fake -55% single-day drop; something like this could be catastrophic to pairs formation for example (or an exit etc.)

The adjustment ratio stays constant throughout the event (up to rounding). What actually happened was that holders received one Wyndham Hotels share per share held, and WH closed its first session at 61.40, so the true outcome was (48.71 + 61.40) / 108.44, or +1.54%.

The second issue was with TT, which turned out to be the most useful failure case. Here, the event was present but not fully applied. Ingersoll-Rand plc became Trane Technologies and distributed 0.8824 shares of the new IR per share in a reverse Morris Trust, a specialized corporate finance strategy under U.S. tax law that allows a parent company to spin off a subsidiary or unwanted business asset and merge it with a third-party company completely tax-free. Between 2020-02-28 and 2020-03-02 the raw close moved 129.04 to 100.55, a return of -22.1%, while the stored adjusted close moved 104.72 to 92.81, or -11.4%. Counting the 0.8824 IR shares received at 32.80, the true outcome was +0.35%. Roughly half of what the event required was applied, leaving about 11.7 points of phantom loss.

Both these events sit inside an eligible cohort window. All eleven events end up characterized as follows, and SNDK is terminal: the series ends at the acquisition and there is no seam.

Internal verification checks were applied to the series produced. This fix means that no event carrying a nonzero cash or stock term may be left to Alpaca or to just an identity split. A nonzero distribution also has to force an explicit stitch. We checked this over the whole dataset and also if any other inconsistencies were present in spin-offs that did not result in these convoluted name changes. See the above entries.

After lots of pain, I discovered the root cause of many of the issues. It seems that the public membership data given by the GitHub uses back-labelling in lots of places, meaning it uses a ticker in an early sampled date that a company only acquires later in its history. A lot of corporate-actions nonsense exacerbates this issue or even creates it, and these were initially not resolved with Alpaca asof fixes (we used asof for the latest cohort dates, meaning we said to Alpaca “give me the full price history for the company labelled by this ticker on the latest date this ticker appears”), which is of course almost guaranteed to screw up for securities with these renaming histories and such.

An illustrative example is Delphi Automotive. In 2016 it was trading under the ticker DLPH; in December 2017 it spun off Delphi Technologies as a child (which then took the DLPH ticker with it when it spun off) and relabeled itself to Aptiv PLC, and started using the ticker APTV. In the membership list, the 2016 sampled date shows APTV (which is not possible); when using Alpaca for APTV with any asof date, the price history begins on 2017-11-17, which is when that ticker first started being quoted. The first few weeks of that series are still the pre-separation parent, with bars identical to DLPH’s up to 2017-12-04, and it becomes the skinnier company from 2017-12-05.

The fix with these kinds of events is economic; clearly there are two unique price series here, one for the company that relabels itself from DLPH to APTV, and another that begins its life as DLPH. Only the first is needed here, since Delphi Technologies was never an S&P 500 constituent after the separation and so appears in no membership list. The original company’s adjusted price series has to remove the apparent loss at the separation, because the fall in the raw price is offset by the one-third DLPH share the holder received, and its full raw price series will be used for execution. This is clearly the correct treatment for this situation. In terms of labelling, I think it makes sense to use APTV for the company that went through the label change for its entire history, and use DLPH for the company that began life as DLPH. This is internally consistent, and fine for our purposes.

11 such similar issues were found and corrected, listed below. The other unique issue is illustrated by SanDisk, where the same label was being used by two different entities: the SanDisk that was acquired by Western Digital in 2016, and the new SanDisk that listed in 2025, which spun off from Western Digital itself into the semiconductor industry and began using its legacy ticker symbol.

Child takes away the parent ticker. Five of the eleven flags arise this way. The company keeps trading but adopts a new ticker while the separated child inherits the old one, so the old string carries on looking continuous even though it now belongs to a different company.

• APTV: Delphi Automotive spins off Delphi Technologies, renames to Aptiv and takes APTV; the child takes DLPH. DLPH through 2017-12-04 joined to APTV from 12-05, carrying one third of a DLPH share. Raw -14.9%, adjusted +3.17%. Source
• ARNC, 2016: Alcoa Inc. splits; the parent renames to Arconic and takes ARNC while the new Alcoa Corporation takes AA. AA through 2016-10-31 joined to ARNC, carrying one third of an AA share. Raw -34.1%, adjusted -7.4%. Source
• ARNC, 2020: the same company does it again. Arconic renames to Howmet and spins off a new Arconic Corporation which takes ARNC. Carries one quarter of an ARNC share. Raw -17.8%, adjusted -7.0%. This is why ARNC appears twice, and why eleven events cover ten labels. Source
• WYND: Wyndham Worldwide splits into Wyndham Destinations (WYND) and Wyndham Hotels (WH). WYN through 2018-05-31 joined to WYND, carrying one WH share per share. Raw -55.1%, adjusted +1.54%. Source
• IR: Ingersoll-Rand spins its Industrial segment into Gardner Denver, which renames to Ingersoll Rand and takes IR; the parent renames to Trane Technologies. The only case needing a cohort split rather than a join: the first nine cohorts map to TT, the remaining thirteen to the new IR, with 0.8824 IR shares in TT's seam. Raw -22.1%, adjusted +0.35%. Source

Mergers where the ticker survives but the entity changes.

• BHGE to BKR: Baker Hughes Incorporated combines with GE Oil & Gas; holders receive one BHGE share plus $17.50 cash, and BHGE later renames to BKR. BHI through 2017-07-03 joined to BHGE/BKR with the cash in the seam. Raw -35.4%, adjusted -5.1%. Source
• FTI: FMC Technologies merges with Technip and TechnipFMC keeps the FTI ticker. Not back-labelling, since FTI was genuinely FMC's ticker, but the latest-asof request returned only the post-merger entity. An early asof returns the whole lineage and no stitch is needed. Raw and adjusted both -1.4%. Source
• AABA: Yahoo sells its operating business to Verizon and the residual holding company renames to Altaba. YHOO with an early asof returns all 944 rows across both eras. No consideration, because the company sold assets and kept the proceeds rather than distributing anything. Raw and adjusted both +3.6%. Source

Same holder, different registration.

• XL: a redomestication from Cayman to Bermuda. Legally two entities, economically one, and Alpaca indexes them separately. Joined with no consideration; raw and adjusted both +0.18%, which is the correct signature for a pure redomestication. Source
• UAA: Under Armour issues Class C. Class A moves from UA to UAA and Class C takes the freed UA ticker. Class A history through 2016-12-06 joined to UAA, taking care not to pick up the Class C now sitting on UA. Raw and adjusted both +4.3%. Source

Terminal identity.

• SNDK: SanDisk is acquired by Western Digital in 2016 for $67.50 cash plus 0.2387 WDC per share, and in 2025 a new SanDisk spins out of Western Digital and takes the ticker back. No join at all here: SNDK_2016 terminates on 2016-05-12 and is never connected to the 2025 series. Two separate price series, and the only case where the fix is to keep things apart rather than stitch them. Source

Eleven events across ten membership labels, covering 102 of the 11,092 cohort rows. Seven further events were added later for DOW, DD, LB, CHK, DO, ESV and MNK, which came out of the tradability pass and are covered in the entry above.

While cleaning the price data we also identified internal issues with the Alpaca data itself. The first is that it does not necessarily track all name changes: the corporate-actions endpoint (which we use to get dividends and split data) includes a name_changes record type; however, I discovered that there was missing dividend data for 9 names. I ran a check on the original 515 constituents (the current ones) by dividing the raw price series by the adjusted series. This is a piecewise constant series, with jumps on ex-dividend dates. I then compared the jump timestamps and sizes against the data given by the corporate-actions endpoint (after correcting for penny-rounding noise, and for spin-offs filed only under the child symbol rather than the parent being tested) and discovered the discrepancy. Through comprehensive searches I was able to resolve the discrepancies, e.g. Google searches for press releases and SEC filings, mostly on those dates and for those amounts, and was able to ascribe the dividends to names associated with the company but not tracked on Alpaca. The following table lists the name changes obtained in this way. Note that the more comprehensive headache this reveals will have to be healed when we obtain (and inevitably clean) the corporate actions data.

Note that the DowDuPont line was a merger followed by two planned spin-offs. Dow Chemical and DuPont combined into DowDuPont in August 2017, the combined company reorganized itself along business lines, and then split into three: Dow Inc. was distributed in April 2019, Corteva in June 2019, and the remainder was renamed DuPont de Nemours. All of this was the announced plan. It is the reason DOW is used by two different entities in two different periods, the old Dow Chemical until August 2017 and the new Dow Inc. from April 2019, with nineteen months in between where nothing traded under the ticker at all. This is the sort of business nonsense that the data is riddled with and had to be cleaned.

Querying those earlier labels by hand recovered 122 cash dividends; note that it seems Alpaca still holds the dividend data, but that how it serves price data (aggregating the symbols a security has used when a history is requested) differs from how it serves corporate-action data (tied to one particular symbol). This discovery will probably help us recover all the corporate actions data once we begin collecting it, but for now, we will still focus on cleaning the price data.

The first issue in the cleaning process is that once we obtain the point-in-time membership lists at the sampled dates (30 June and 31 December of each year, running from 2015-12-31 through 2026-06-30 for 22 cutoffs in total), the lists contain the stock ticker symbols used by companies on that date (e.g. 2016-H1 contains FB for Facebook and 2023-H2 contains META). The distinct labels produce 738 symbols (not counting the 12 ETFs).

This causes multiple issues, since companies use multiple labels over their history; to make matters worse, each label may (and is) also used by multiple companies when tickers are freed up, and a company may use the same label multiple times as well. Tickers can die and come back to life, and refer to the same or a different company as well, and a label can outlive a company. This means the ticker labelling process is also coupled to the bankruptcy, delisting, merger/acquisition, spin-off process! The headache cannot be overstated.

Fortunately, the full ramifications are not felt for our purpose; the labelling in a particular cohort is arbitrary to us; we don't care if we call a particular price history inside a particular cohort FB or META; it just needs to exist and be unique among its own cohort. However, I ended up annoyed enough by all this to resolve exactly each individual event and produce fully cleaned data, and document all of the resolutions.

Alpaca's API lets us call using the asof parameter; when we supply a date to it, e.g. asof=2021-12-31, it returns the full price history for the security with the passed symbol as of the supplied date. This is of course extremely helpful; we pass each of the given symbols in the 22 cohort lists with the appropriate asof date i.e. the date of the cutoff for that symbol (FB last appears in the 2022-H1 cohort, so it is requested with asof=2021-12-31, while META, which is still a member, is requested with asof=2026-06-30); note that this returns the full history for that security, not just in the given period.

To illustrate why this is useful, if we call FB without any asof, we get a spliced history of two companies joined together: Facebook until they renamed, then a three year pause, and then a different issuer that later claimed the freed up FB ticker. The pause is not stored as empty or missing values; the bars are simply absent. The returned series has 1,896 rows, of which 1,620 run from 2016-01-04 to 2022-06-08 and the remaining 276 begin on 2025-06-26, leaving a 1,114 day hole and a fabricated move from 196.64 down to 39.91 across the join. Using META with no asof returns the exact same data as FB with the correct asof date, which is why we have duplicate price series after the nominal data requests for the full 738 symbols.

To identify duplicate series, we compare the unadjusted price series against each other (because adjusted values can differ, as the corporate-action history attached to the chosen symbol identity and asof basis can differ). We then retain the latest label as the canonical storage label (we preserved the query responses as the raw data as well).

Note that each ticker may appear in multiple cohorts, and we did not re-run the request for each instance of a symbol. Instead we use the appropriate asof date associated to the latest cohort the symbol appears in, and then deduplicate by comparing price histories; this worked well for recognized ticker changes e.g. the FB to META change described above. Note the comparison was made by matching hash fingerprints for the series (byte-identical returns from Alpaca), and we were able to identify the 30 “extra” tickers to drop that appeared due to name changes. 3 of the companies went through a name change twice, meaning each of these had 2 discarded labels rather than 1.

However, this does not work for all cases. One failure is retrospectively relabelled memberships, and incomplete Alpaca name change documentation, e.g. APTV appears in every membership cohort beginning 2016-H1 but the Alpaca history begins on 2017-11-17 (same issue with AABA, which appears from 2016-H1 but whose history begins 2017-06-19). When Alpaca cannot match the asof parameter, it simply skips the name change process, and returns the data belonging to the passed string. The other issue is ticker reuse, e.g. SNDK (currently used by SanDisk) which appears in 2016-H1, then again in 2026. Since we made the request using the latest cohort, we got the full history of SanDisk, but this is not the company that was using this symbol in 2016 (since the history only begins on 2025-02-13).

I have now updated the entire universe to include the price history (adjusted, unadjusted) for all 720 symbols (later updated to 724, see the identity repairs below). The horizon window remains 2016-01-04 until 2026-07-31, and the data overwrote the existing 515 store for simplicity and better organization.

Out of the 750 symbols, 30 symbols that appeared in the membership list did not generate an Alpaca price history from the requests, all of them due to the fact that these symbols were prior ticker labels used by companies (e.g. FB became META); this means that the successor (latest) symbols are the ones we use to access the price history and the total unique price symbols are 720 (including 12 ETFs; later 724, once reused tickers such as SNDK were separated into distinct securities).

I decided after discovering existing data sources e.g. fja05680/sp500 and hanshof/sp500_constituents that provide point-in-time membership of the S&P 500 that we ought to try to get rid of the survivorship bias introduced by using the current index constituents, which was reinforced by the fact that since 2016 about a third of the index has been reconstituted so the bias would be quite strong.

Using the GitHubs, I was able to get semi-annual interval membership lists (sampled June and December) and assigned them to be the canonical universe; each “cohort” acts as the active universe in its own walk-forward six month backtesting period, and we roll this forward across the entire backtest horizon up until the holdout period. The itemizing and tabling of this data was quite the mess; I cross-checked the data with the existing current membership data that we sourced manually using Wikipedia and SlickCharts, and also cross checked it with the secondary GitHub (which was somewhat less complete). The ticker symbols were also normalized to follow the Alpaca dot convention; this had caused an issue in the previous project where Berkshire Hathaway used BRK-B and BRK.B in different sources for example. The data is now fully itemized, available in human-readable CSVs, and cleaned. The exact configurations, conventions and code are more appropriate for the GitHub.

The primary source is the MIT-licensed fja05680/sp500 historical components dataset. It combines an older historical base with manually maintained later changes and documents the need to supplement Wikipedia's incomplete change table.

The MIT-licensed hanshof/sp500_constituents dataset is retained as a secondary audit source. It increasingly agrees with the primary source in recent years, but it has omissions earlier in the study, two dates with conflicting duplicate rosters, and one annotated value that is not a valid ticker.

Getting price data and especially corporate-action data, then subsequently cleaning them will now be the next headache.

The text below is the original Data section of the project page, kept as a record of the earlier approach before the walk-forward cohort design replaced it. It described a universe taken from index membership as of the fetch date and backfilled, and proposed accepting the resulting selection bias and stress testing it with synthetic data instead of removing it.

“The price data was sourced from Alpaca. The attributes I decided on for this project are listed below, and the asset names are listed here. The primary compromise made is that the equity universe is names belonging to the market index today; their historical price time series was backfilled until 2016; over this period, a large reconstitution is expected to have taken place, and the largest risk associated is that stocks can A) be delisted from the index, B) go bankrupt entirely (i.e. price goes to 0). These and similar situations (e.g. mergers, acquisitions, etc.) are largely absent from the data, and since pairs trading algorithms largely bet on convergence, the risk associated with these events is not represented in the data, and in general this also biases the sample upwards (the universe consists of companies that will perform well through the time-series). This is future-membership lookahead structurally, and in particular also biases tail risk towards 0.

“This selection bias must be accepted if one does not use point-in-time data. Note that we may have access to this eventually through WRDS/CRSP, but for this current study, we restrict ourselves to a feasibility analysis only, and present results with this in mind. Since we are not, for now, we will need to design and implement stress testing and also perform more thorough risk management for the aforementioned regimes, in particular, catastrophic divergence. This will require synthetic data generation.

“Refer to the log for some issues that were encountered with the original data fetching process and stores, and how they were fixed and accounted for in future requests.”

Universe

515 tickers: 503 S&P 500 constituents, plus 12 benchmark and sector ETFs

Frequency

Daily OHLCV bars

Coverage

2016-01-04 – 2026-07-31

Adjustment

Adjusted and unadjusted, stored separately

Size

1,030 Parquet files (515 per adj.), 1,320,582 rows per adj.

Completeness

472 of 515 series per adj. span the full 2,659 sessions

Retrieved

2026-08-03

514 out of the 515 securities in the universe are tradeable (easy to borrow, shortable, and marginable). One symbol (EA) was discovered to be untradeable because it was delisted, caused by an acquisition of EA as announced here on 2026-08-04. Since it has price history and was tradeable in the backtest horizon, which ends 2026-07-31 and so closes before the delisting, we will still use EA in the universe and proceed normally.

Identified an issue with the retrieved data. A stock is listed twice with two separate ticker symbols; this was probably caused by the following stock symbol change to EchoStar from "SATS" to "ECHO" which occurred on June 24, 2026, during the data retrieval process; the constituent list carried both symbols with the fields split between them, which surfaced as missing values while generating the universe listing for this site, and the time series was verified to be bit-identical on the overlap period. We drop the stale SATS series and keep the back-filled full history under the new ticker ECHO. Since these two series were exact duplicates, this error would have produced a perfectly cointegrated pair, but completely untradeable and would have produced an erroneous pair ranking.

This issue also led us to another possible consideration: that of dual-class listings, and we also conducted a larger sweep to identify identical or constant multiple series to ensure the veracity of the data.

Fixed. The specific cause was a disagreement between the two sources the constituent list is built from. Rank, company name and index weight come from SlickCharts, while GICS sector and sub-industry come from Wikipedia, and the two are combined with an outer join on ticker. At the time of the scrape SlickCharts still listed the company under SATS while Wikipedia had already updated to ECHO, so instead of one complete row the join produced two half-populated ones, and both symbols were then fetched into the store. The two constituent rows were merged into one under ECHO, leaving 503 constituents; the stale SATS series was moved out of the store; and a duplicate-detection check was added that runs automatically at the end of every fetch and refuses to record a snapshot if two tickers hold identical price series. The join itself was also fixed: it now reconciles a symbol change across the two lists rather than emitting two half-populated rows, and raises an error if it cannot. The whole universe and price history were then re-fetched on 2026-08-03, and both sources now agree on 503 constituents with no reconciliation needed, since the symbol change had settled by then.

I identified break points and seams in the adjusted price time series datasets when expanding the coverage from 2020–2026 in the earlier project to 2016–2026 for this project. After further investigation, I discovered that this was caused by the data fetcher design, which first looked up the existing dataset coverage, and expanded it to the request time-period by appropriate back and forward fills to reduce the API calls.

However, this is an issue since adjusted time series are rebased (and this is in fact how I discovered this in the first place), meaning they have a particular date that is fixed as the anchor, and the remaining time series is normalized (e.g. for dividends, stock splits etc.) relative to this anchor (usually, and by default, this anchor is set at the latest/current date, a consequence of which is that the latest price from the broker is what a trader sees live in the market, and the past prices are then adjusted backwards); for multiple, independent calls for disjoint periods, this produces the seams in the series.

It also means an adjusted price for a past date is not fixed, since it changes whenever a later split or dividend occurs. This is not catastrophic in the returns setting (as it produces only a few, constant erroneous values), but it is much more destructive for levels, since the cointegration and trading assumptions rely on the veracity of the levels data.

Fixed. We re-requested the whole 2016–2026 period in one call, so the entire series carries a single adjustment basis, and the store is now always fetched into empty directories rather than merge-updated. This is also where I learned why this project needs both adjustment modes, and why a more complicated trading model and backtester will need them too, which I discuss in a later section.

Apache Parquet is an open-source columnar, binary storage format which is much more efficient on storage and query performance than row-oriented formats like CSVs. Parquet groups the data into contiguous blocks of memory on disk, which is what makes it more efficient. The rows are grouped and each of the column chunk per group is encoded then compressed, while the footer contains the schema and per-column statistics. The schema is the file's own record of what each column is called, what type it holds and how that type should be interpreted (a timestamp with a timezone, say, rather than an integer), so a reader reconstructs the frame from the file instead of guessing from the text as it would with a CSV. The statistics hold the minimum and maximum of each column within each row group.

An example for us is the AAPL daily file: 2,659 rows over six columns in a single row group, Snappy compressed, 107 KB on disk compared to 166 KB for the CSV version. The columns it holds are below.

The format is read and written by PyArrow which is the official, high-performance Python library for Apache Arrow, the larger data-handling/storage framework and columnar layout; PyArrow is the Python binding to the C++ implementations. Historically, Pandas relied entirely on NumPy as its internal engine, but since pandas 3.0 PyArrow is a required dependency. We are on pandas 3.0.2 with PyArrow 24.0.0 under Python 3.13, so handling the parquet files is just done natively with pandas!

The research universe page is kept as a record of the earlier approach and is no longer part of the live dataset. It lists the S&P 500 constituents with their weights, sectors and sub-industries as scraped from Wikipedia and SlickCharts on 2026-08-03, and the price history was then obtained backwards for exactly that roster.

That is the construction the point-in-time pivot replaced. A roster taken on one date in 2026 and backfilled to 2016 carries future-membership lookahead: it contains the companies that survived to be members in 2026 and excludes those that were delisted, acquired or went bankrupt along the way. The live universe is instead the union of 22 semiannual membership snapshots, each frozen at a cutoff before the cohort it governs, which is what cohort_price_identities.csv encodes.

The page is retained rather than deleted because the weights and sector labels on it are a real 2026-08-03 observation, and because the reasoning that led away from it is part of the record. It should not be read as a description of the universe this project trades.