Feature Request: LOCAL Series – Import and Sync CSV Files from a Selected Folder

Suggestions · tradewithmadhan,

Context

First, thanks for implementing [[#266](https://github.com/)](https://github.com/) — the ability to import CSV bar files as LOCAL:<name> series is a great feature.

Currently, imported CSV files are stored in the application's data folder on Desktop (and per-tab in the Web demo). Importing data requires opening the Import CSV dialog, manually selecting multiple files, previewing them, and importing them.

The problem is that whenever the source CSV files are updated, we have to reopen the dialog and select the same files again.

In real-world workflows, market data is usually maintained in folders containing hundreds or thousands of files, such as:

- NSE/BSE bhavcopy collections.
- EOD vendor daily exports.
- Historical intraday data.
- Research datasets generated by external scripts or spreadsheet pipelines.

This is similar to the classic AmiBroker data-folder workflow, where the application monitors or periodically scans a configured data directory.

LOCAL series are already a natural chart target for this workflow. The missing feature is automatic folder-based importing and synchronization.

Proposed Feature

Allow users to register one or more Local Series Folders that the application can import and synchronize automatically.

1. Import Folder (Initial Step)

Add an Import Folder button to the existing Import CSV dialog.

Expected behaviour:

- Allow users to select a folder.
- Import all top-level .csv files.
- Also detect and import symbol_info.json, if present.
- Display import progress while processing files.
- Process files sequentially, one at a time:

Read → Parse → Save → Release Memory

- Avoid loading hundreds or thousands of files into preview memory simultaneously.
- Display a final summary:

- N files imported successfully.
- M files failed or encountered problems.
- Provide individual error reasons for each failed file.

Additional improvement to the existing import dialog:

Currently, one invalid file (for example, an accidentally selected README or malformed CSV) can disable the Import button for the entire batch.

It would be better to support a per-file Skip action so that one invalid file does not prevent the remaining valid files from being imported.

2. Automatic Folder Synchronization (Main Feature)

Add a Local Series Folders section under Settings.

Users should be able to register multiple folders and configure their synchronization behaviour.

The backend should synchronize registered folders automatically.

When to synchronize:

- On application startup: Start synchronization in the background after the application becomes idle.
- After a session closes: Rescan registered folders and process any new or modified files.
- Alternatively, support a lightweight periodic scan or filesystem watcher.

Synchronization logic:

For every registered folder:

1. Scan the folder for CSV files.
2. Compare each file's modification time and size (or content hash) against its previously recorded state.
3. Skip unchanged files.
4. Parse only new or modified files.
5. Upsert the corresponding LOCAL series using the existing local_series_save functionality.
6. Record the latest file state after successful processing.

The synchronization must be sequential and memory-efficient.

Important: An unchanged library containing approximately 3,000 CSV files should require only a directory scan and metadata comparison, not a complete re-import or re-parse.

3. Data Model Changes

Introduce a backward-compatible source field in LocalSeries.

Suggested structure:

- source: null – Manually imported series; never managed by folder synchronization.
- source: { path, mtime, size } – Series associated with a registered folder.

Expected behaviour:

- Existing manually imported series remain untouched.
- If a folder contains a CSV with the same name as an existing manually imported series, importing it should adopt that series as folder-managed.
- Subsequent synchronization should update the same LOCAL series rather than creating duplicates.
- Track the source file's metadata to determine whether re-import is necessary.

4. Optional File Deletion Handling

Provide an opt-in setting:

Remove LOCAL series when their source files are deleted from the registered folder.

- Default: OFF.
- When disabled, deleting or moving a source CSV must not delete the corresponding LOCAL series.
- When enabled, series managed by that folder can be removed when their source files no longer exist.

This should be conservative by default because users may reorganize or move historical data files without intending to delete the stored chart data.

5. Synchronization Status and Reporting

Under Settings, show the last synchronization status for each registered folder.

Suggested information:

- Last successful sync time.
- Number of files scanned.
- Number of files imported or updated.
- Number of bars processed.
- Number of skipped files.
- Number of errors, with individual file-level reasons.

Reuse the application's existing notice/notification surface to report synchronization results and errors.

Implementation Considerations

The existing CSV import implementation already has the required parsing and saving functionality.

Ideally, folder synchronization should reuse the existing pipeline:

CSV File → Existing Parser → local_series_save → LOCAL Series

There should be no need to introduce another bar-reading or CSV-parsing implementation.

The main additions would be:

- Folder registration and configuration.
- Background synchronization scheduling.
- File metadata tracking.
- Source ownership tracking.
- Incremental synchronization.
- Progress and error reporting.

Platform scope:

Desktop should be the primary target because its application data folder and LOCAL series have a persistent lifetime.

The Web demo can either skip automatic folder synchronization initially or optionally support it using the File System Access API, with a directory handle retained within the browser tab.

Testing

I have a real-world library of approximately 3,000 CSV files and would be happy to test an early implementation or prototype.

My dataset includes multiple formats supported by #266:

- Daily OHLCV data.
- Intraday OHLCV data.
- Different CSV column formats.
- Single-value columns.
- A large collection of historical market data files.

Testing should cover:

- Initial bulk import.
- Incremental import of newly added files.
- Updating an existing CSV.
- Skipping unchanged files.
- Handling malformed or unsupported files.
- Restarting the application and resuming synchronization.
- Optional deletion handling.
- Memory usage with thousands of files.

Expected Outcome

The goal is to make LOCAL series work like a traditional data-folder-based charting system.

Users should only need to configure their data folder once.

After that:

- Drop new CSV files into the folder.
- Update existing CSV files whenever necessary.
- Open the application.
- The application automatically detects and synchronizes the changes.
- Charts using the corresponding LOCAL series always reflect the latest available data.

The folder becomes the single source of truth, eliminating the need to repeatedly open the Import CSV dialog and manually re-import files.