Data Processing System Design
Data Processing System Overview
Data Processing systems read source data as input, such as customer transactions, internal transactions, external data feeds, and real-world measurements. The source data is cleansed, loaded into the Data Processing system (usually into a database), then is transformed into the format desired for further processing. Once the source data is combined with data from other sources, one can correlate, compare and contrast the data to help answer business questions, or generate new data products.
Foundation Software Technology has experience with designing, implementing and enhancing several different types of Data Processing systems. One such system reads data from customers over the Internet, which is then loaded and transformed into a Data Warehouse. Since the data feed gathered from the customers can change as the business needs change, the Data Processing system must be able to adapt to the changes in the data stream, while still retaining the ability to process older data formats.
Foundation Software Technology has also worked on enhancing Data Processing systems that are used in the manufacture of pharmaceuticals. These systems are FDA 21 CFR Part 11 validated systems, and must comply with FDA Good Manufacturing Processes. All changes to these systems must be documented and tracked in order to prove to FDA auditors that FDA regulations have been followed.
Foundation Software Technology also has experience designing and implementing Data Processing systems used in manufacturing. The system read product counts and barcode scans from the factory automation, test results from automated test equipment, and user input from Quality Control Inspectors and Analysts, and stored them in the database. From this database, production management could produce reports and graphs that show how well the production systems were functioning. Analysts could retrieve historical test data for a product that failed a test to determine whether the product was salvageable. Automation technicians could use the historical test data to determine whether production machinery was failing or drifting out of calibration.
Foundation Software Technology has also worked on Data Processing systems used to provide military and commercial Flight Simulators with the data used to create the realistic graphical displays viewed in their windows. This data is first entered with human-friendly editors to model the terrain, cultural objects, and graphical textures. Then the data is processed and transformed into the low-level hardware-formatted data that is used by the graphics processors in real time.
What to Look For in a Data Processing System
What are some characteristics of a good Data Processing system? Here are some characteristics to look for when evaluating or designing a system.
- Processing configuration tracking and management, allowing one to know exactly the version and configuration of the software used to process each bit of data.
- Data feed change management to allow the system to adapt to future data feed changes, while retaining the ability to process older data.
- Design for Administration, to facilitate backups, disaster recovery, day-to-day system management, and data archival.
- Software extension points to allow for future software changes easily.
- Documentation of the Data Processing system and its components. Documentation of a data processing stream would preferably be built using the current software configuration metadata.
Foundation Software Technology has experience with complex Data Processing systems, from Flight Simulators to Market Research systems. If you have the need for the design and creation of a new Data Processing system, or the modification of an existing system, please contact us to find out how we can help you.