![]() return a Hash for each line of the CSV file, so we can quickly use the results for either creating MongoDB or ActiveRecord entries, or further processing with Resque.able to chunk the input from the CSV file to avoid loading the whole CSV file into memory.Smarter CSV was a big part and helped clean up our code ALOT taking an import process from 7+ hours to about 3 minutes. You can use the options below to accomodate for unusual formats. If you encounter unusual CSV files, please follow the tips in the Troubleshooting section below. Therefore, when calling SmarterCSV.process, please rescue from SmarterCSVException, and handle outliers according to your requirements. ![]() While SmarterCSV uses sensible defaults to process the most common CSV files, it will raise exceptions if it can not auto-detect col_sep, row_sep, or if it encounters other problems. ![]() formatting each row of data as a hash, in order to allow easy processing with ActiveRecord, parallel processing, kicking-off batch jobs with Sidekiq, or oploading data to S3.improve robustness of your code when you have no control over the quality of the CSV files which are processed.ease of use for handling most common CSV files without having to tweak options.Smarter_csv is a Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, suitable for direct processing with ActiveRecord, parallel processing, kicking-off batch jobs with Sidekiq, or oploading data to S3. Please check the 2.0-develop branch, open any issues and pull requests with mention of tag v2.0. Work towards SmarterCSV 2.0 is still ongoing, with improved features, and more streamlined options, but consider it as experimental at this time.2.x development is on 2.0-development (check this branch for 2.0 documentation).default branch is main for 1.x development.SmarterCSV Development Branches Work towards Future Version 2.0 SmarterCSV 1.x Rescue from Exceptions Features Why? How? Troubleshooting Examples Example 1a: How SmarterCSV processes CSV-files as array of hashes: Example 1b: How SmarterCSV processes CSV-files as chunks, returning arrays of hashes: Example 1c: How SmarterCSV processes CSV-files as chunks, and passes arrays of hashes to a given block: Example 2: Reading a CSV-File in one Chunk, returning one Array of Hashes: Example 3: Populate a MySQL or MongoDB Database with SmarterCSV: Example 4: Reading a CSV-like File, and Processing it with Sidekiq: Example 5: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV: Example 6: Using Value Converters Parallel Processing Documentation Options: Deprecated 1.x Options: to be replaced in 2.0 NOTES about File Encodings: NOTES about CSV Headers: NOTES on Duplicate Headers: NOTES on Key Mapping: NOTES on the use of Chunking and Blocks: NOTES on improper quotation and unwanted characters in headers: See also: Installation ChangeLog Reporting Bugs / Feature Requests A Special Thanks to all Contributors! □ □ □ Contributing README.md
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |