Semi-structured data

Machine-generated sources

Machine-generated semi-structured data sources are automated systems and devices that output information organized with tags, keys, or markers rather than rigid tabular schemas. Examples include server log files, IoT sensor metrics, event streams, NoSQL database engines, email systems, mapping, and geographic information systems.

System and server logs

Application outputs, error logs, and security audit trails can use text patterns or key-value pairs without a strict schema.

IoT and telemetry streams

Many smart devices, industrial sensors, and interconnected hardware components generate periodic metrics packaged in JSON or XML.

Web and API activity

Automated webhooks, clickstream tracking data, and web API response payloads are typically packaged in JSON.

Network & monitoring data

Packet inspection summaries, router statistics, and application performance management (APM) metrics provide semi-structured data in various formats.

NoSQL database engines

A NoSQL database management system, often used in big data and real-time applications, is designed to handle semi-structured data models, including key-value, document, column family, and graph formats

Email headers

An email consists of headers and a body. An email header conforms to a standard key-value format that includes a field name followed by a colon that serves as a delimiter.

Mapping and geographic information systems

GeoJSON is a standard for encoding geographic data structures, such as points, lines, and polygons, into a JSON-based format used to represent geographic features, such as locations, roads, boundaries, and their names. OpenStreetMap stores semi-structured data using a flexible tag-based data model built on elements such as nodes, ways, and relations.

Formats for machine-generated semi-structured data

Application-specific or domain-specific formats

Domain-specific datasets are collections of data specific to a particular field. These datasets focus on specific types of information relevant to a domain such as health care or finance. The data model or format used for domain-specific data may be tailored for use cases within that domain.

An application program or a database management system may work with semi-structured data without regard to its interoperability with other software, but a semi-structured data model that exists for its own sake may be unnecessarily cumbersome. Such systems have been designed so that data associated with object-oriented software is serialized into a semi-structured format, allowing for the concept that objects of a class may have varying attributes. The data associated with the objects of a class may be grouped, and the attributes' order is not important. In some object-oriented database management systems, such as ObjectStore, the format for semi-structured data storage is specific to the system.

Database management systems that lack interoperability have fallen out of vogue. Standardized formats for semi-structured data have arisen to serve as a basis for interoperability, for scalability, and for adaptability to schema evolution.

XML and OEM

In the 1990s, XML, and the Object Exchange Model (OEM) were created as generalized formats for semi-structured data. XML has been popularized by web services developed utilizing SOAP principles of decentralization, transport independence, and XML-based structured messaging. XML suffered from the impression that it lacked structural rigor at the same functional level as relational schemas. Once JSON became popular, its compatibility with JavaScript and its relatively human-readable format handicapped the widespread use of XML. Nevertheless, an XML document can be as rigorously structured as a database schema and processed by both commercial and custom software programs. XML data can have a reasonably human-readable flow and hierarchy as well as rigorous relational structure and data typing.

The concept of XML as human-readable, however, can only be taken so far. Some implementations, such as the XML representation of the contents of a Microsoft Word document, as implemented in Office 2007 and later versions, utilize dozens or even hundreds of different kinds of tags that reflect a particular problem domain - in Word's case, formatting at the character and paragraph and document level, definitions of styles, inclusion of citations, etc. - which are nested within each other in complex ways. Understanding even a portion of such an XML document by reading it, let alone catching errors in its structure, is impossible without a deep prior understanding of the specific XML implementation, along with assistance by software that understands the XML schema that has been employed. Such text is not "human-understandable" any more than a book written in Swahili, which uses the Latin alphabet, would be to a reader who does not know a word of that language but recognizes only its alphabet. Because XML tags express shared vocabularies, they are symbols that are meaningless to a person unfamiliar with the domain in which they apply.

YAML

YAML (YAML Ain't Markup Language) is a human-readable data serialization format designed for data exchange between differing programming languages. It is used for configuration files and for representing complex nested data structures.

TOML

TOML is designed as a configuration file format for mapping file elements to a hash table. Implementations are available for over thirty programming lanauges.

INI

The INI format arranges key-value pairs in named sections. It is used in configuration files for applications that run on Windows and other operating systems.

Log file formats

Log files are used for a range of purposes such as performance monitoring and debugging. They may include semi-structured data. Various log formats are in common use.

Apache Avro, Parquet, and ORC

The Apache Software Foundation provides open source file formats for data storage and retrieval. Each format is designed for a specific data orientation. Apache Avro is a row-based format used for streaming data and append-only logs. Apache Parquet is a column-based format used for data lake repositories. Apache ORC is an alternate columnar data format used for data warehouses.

Delimiter-separated values

Files containing text delimited by newlines, commas (as in a format known as CSV), semicolons, tabs, or other separators are used in data exchange, analytics, and integration.

JSON

JSON (JavaScript Object Notation) is an open standard format that uses human-readable text to represent and stream data objects. JSON has been popularized by web services developed using REST principles. Databases such as MongoDB and Apache CouchDB work natively with data stored in JSON format.

Pros and cons of semi-structured data formats

A database or standalone application may store or stream data in a succinct machine-readable format, or it may use a semi-structured format. Compared to a succinct format, a semi-structured format has certain advantages and disadvantages.

Advantages

• Developers persisting objects from their application to a database do not need to worry about object-relational impedance mismatch, but can often serialize objects via a light-weight library.
• The data representation has flexible schemas, self-describing tags or delimiters, and nested hierarchies, rather than any rigid format, which may simplify data models.
• Support for lists of objects, as needed for object-relational mapping, simplifies data models by avoiding messy translations of lists into a relational or graph data model.
• Choosing a commonly-used human-readable format over a succinct format may aid software debugging and maintenance, interoperability, and schema evolution.
• A semi-structured data format promotes rapid application development via the schema-on-read paradigm by letting applications store data without first modifying a database schema, offering flexibility, fast iteration, and ease of changing the data model.

Disadvantages

• The traditional relational data model and the graph data model have expressive, popular declarative query languages including SQL and Cypher, are built on database engines that rely on binary storage and streaming formats such as Apache Arrow for performance, transactional consistency, and secure access control. Semi-structured data must be transformed for effective use with a relational or graph data model.
• The schema-on-read paradigm raises data quality management and performance overhead concerns.
• Reliance on a semi-structured data format such as JSON, which provides no guarantees about structure, is prone to "garbage in, garbage out"; by removing restraints from the data model, there is often less forethought than is necessary for software profitability.
• Ingesting semi-structured data into a strictly structured relational or graph database can create a clash of architectures unless steps are taken to merge data models.

Query processing for semi-structured data

Merged data model

Some relational databases provide data models in which structured and semi-structured data can be queried at once, typically by incorporating the semi-structured data via data extraction techniques. For example, Microsoft SQL Server has a native JSON type backed by string manipulation, and it can create a B-tree index for nested keys. The Snowflake data platform has methods for loading semi-structured data elements in a columnar format for use with queries. PostgreSQL and AWS Glue support semi-structured data via upstream SQL views, which act as virtual tables used to extract and standardize the data before it reaches the core relational data model.

Semi-structured data model

When semi-structured data is not merged into a structured data model, a query method directly targeting it can be selected depending on the data's expected use. Some database management systems are designed to work primarily with a semi-structured data model. Pattern recognition methods can be applied to semi-structured data via machine learning systems or as declarative programming functionality developed in the context of imperative programming.

The semi-structured data model or schema-less data model used by NoSQL databases entails no separation between the data and the schema, because each record has a self-describing structure. The amount of structure used within a specific data model depends on its purpose, and structural validation may be shifted from the database engine to the application logic.

The advantages of this data model are the following:

• It can represent the information of data sources that cannot be constrained by schemas
• It provides a flexible format for parallel computing involving different types of databases.
• It can be helpful for visual data representation, for example in a web browser.
• It accommodates change, allowing for dynamic or evolving data structures.
• The data transfer format may be portable, allowing for data structures be shared between differing systems and programming languages.

The primary trade-off associated with a semi-structured data model is that queries cannot be made as efficiently as in a more constrained structure, such as in the relational model. Navigational methods for a semi-structured database, such as methods using the XPath expression language, may rely on storing elements with unique IDs. These methods allow for reasonably efficient simple navigational or path-based queries, but for complex searches over many records (as is typical in SQL) they are quite inefficient because they must seek repeatedly through the records.

Shifting aspects of query processing from a database engine to application logic can have drawbacks. Application logic based entirely on imperative programming may not be suitable for the challenges presented by semi-structured data. These drawbacks can be mitigated if the application logic invokes methods suitable for the data.

Deep learning models trained on semi-structured data

Deep learning models can be trained to recognize relationships, metadata, and sequences stored as semi-structured data, in such a way that schemas need not be rebuilt as new signals arrive or as data sources evolve. Human-generated data can be ingested into a machine learning system, such as a deep learning system or semantic search system, via intelligent document processing techniques. Mixed formats that may involve machine-generated data can be combined so that different types of content, structures, encoding schemes, or media types can act as a single file or stream or as a join of records that have a matching key.

Intelligent document processing is a deep learning technique for capturing and processing data from documents such as email, text, word processing, PDF, or scanned documents. Using technologies such as computer vision, optical character recognition (OCR), and natural language processing (NLP), the extracted data can be analyzed, categorized, transformed, and exported to external systems. Deterministic layout-parsing tools like OCR can be combined with generative AI or large language models (LLMs) to process semi-structured documents.

Intelligent document processing for semi-structured data is typically a multi-stage process. The stages may include the following:

• Ingestion and pre-processing collects files (PDFs, scans, images) and cleans visual noise, skewing, or low-resolution artifacts.
• Classification relies on learning or layout models to identify document types, such as by distinguishing a purchase order from an invoice.
• Parsing converts documents into machine-readable data.
• Document layout analysis identifies structural elements of a document, such as text blocks, paragraphs, headings, images, tables, and mathematical expressions, along with their spatial coordinates and order. Layout analysis can involve convolutional neural networks, transformer-based methods, graph-based methods, grid-based methods, and semantic integration.
• Data extraction relies on vision-language models or natural language processing to map key-value pairs, nested tables, and line items into queryable formats. Extracted data can be exported in commonly used formats such as JSON or CSV.
• Data validation evaluates confidence scores on parsed fields, routing low-confidence data to human reviewers for correction and model retraining.

A confidence interval in the context of data extraction is a statistical range that quantifies how confident a system or model is in the accuracy of extracted data. A confidence interval does not guarantee that any individual extraction result is correct, but instead reflects the reliability and precision of the extraction process itself, measured across repeated use.

Intelligent document processing systems rely on the integration of multiple deep learning modules, which can lead to challenges in modular coordination, standardization of outputs, and handling of irregular reading orders in complex layouts. They typically require extensive pre-processing, intricate post-processing, and specialized training for each module to achieve accurate results. They typically depend on rule-based methods, which can be inadequate for documents with complex layouts such as multi-column or nested structures. They often process documents page by page, limiting their efficiency and scalability.

Deep learning models can be applied to interpret formatted semi-structured data, such as JSON or CSV file content from human or machine-generated sources, by automating schema discovery, handling missing values and anomalies, and extracting semantic insights. The methods replace hard-coded rules with learning-based pattern recognition.

To enable schema discovery, deep learning models scan raw text values, for example in JSON fields or CSV columns, to infer whether a given value is a number, date, or text, and to discover whether formatting is inconsistent. Named entity recognition methods and other natural language processing methods can match inconsistently named JSON keys (like "last_name" vs. "surname") to a standard data schema. Regression or classification models can infer missing values based on surrounding data rows or nested JSON attributes. Nearest neighbor search algorithms such as DBSCAN can be used to flag anomalies such as outliers, corrupted rows, or injection attacks within log files or nested API payloads. Large language models can be used to extract named entities and sentiment from text embedded within JSON data.

Software applications and services can utilize the extracted data via any of several methods. Data manipulation toolkits such as the Pandas library can be used to convert JSON data into a tabular format. The resulting data can be stored in a relational database via additional tools such as SQLAlchemy, or it can be accessed directly via imperative program code written in C++ or other programming languages.

The development process for an application that relies on deep learning involves training relevant models on relevant data then integrating the models into user-facing applications. A range of challenges can occur during the process. For example, poor data can undermine machine learning models, and to correct it, developer effort may be directed toward cleaning and validating data. A deep learning model that performs well during training may perform poorly in real-world situations, and to correct this, developer effort may be directed toward cross-validation, regularization, and retesting. The training process can be computationally expensive. Because machine learning models are not static, developer effort may be directed toward building an automated pipeline that collects new data, retrains models based on the new data, evaluates performance, and deploys the retrained models. If an application's developers successfully address these challenges, they can enhance the user experience.

Some deep learning deployments are unsuccessful. Certain datasets are not amenable to deep learning or other machine learning techniques. Insufficent infrastructure readiness can lead to project failure. Data bias, or distortions in datasets that cause machine learning models to develop skewed understanding of patterns and relationships, can create fundamental flaws in how models interpret information. Successful deployment depends on selecting hardware resources based on the machine learning models' computational needs as well as budget constraints. Invalid findings from machine learning systems can be presumed valid, leading to poorly-made decisions on the part of a system's users.

Semantic search on semi-structured data

Semantic search is a machine learning technique applied to concepts and the words that describe them. The concepts are represented in a vector space model trained to recognize similarities and differences between words in a given context. Unlike deep learning models, which can be trained to recognize relationships, metadata, and sequences stored as semi-structured data, semantic search focuses on extracting meaning from data by analyzing search query context, enhancing the relevance of search results. Semantic search applied to JSON data can interpret relationships between fields and entities without requiring extensive training data.

Software applications and services written in imperative programming languages can invoke semantic search methods by generating a vector embedding for a search query and by querying a vector similarity library or database. Typical steps involve preparing a corpus of searchable documents, generating vectors that capture semantic meanings within the documents, building a searchable index, and running semantic queries. An application can invoke semantic search methods to become more robust to paraphrasing or mixed vocabulary compared to relying on simple keyword search techniques, and it may be relatively forgiving of user mistakes.

The computational expense of semantic search methods involves generating vector embeddings for each document and query. When third-party infrastructure is used for semantic search, query costs can scale with database size, and the overall costs can include hidden fees. Maintaining data privacy may require further overhead such as encryption, regulatory adherence, access controls, and monitoring tools.

Declarative functionality within imperative code

Heuristics applied to semi-structured data are practical strategies used to infer implicit structure, to parse data represented in formats like JSON, XML, or CSV, and to accelerate search or extraction without heavy computational expenses. Information extraction from natural language documents involves extracting structured information from unstructured or semi-structured text data. Methods used include string search, full-text search, regular expression methods, delimiter search, and wildcard search.

String search, or searching a text document for a specified string, involves opening the text document file, reading its contents, applying a string-searching function, and closing the file. Many programming languages have built-in string search functionality. The Knuth-Morris-Pratt algorithm is an efficient string-searching method that finds a search term or substring inside a main text by using precomputed prefix information to avoid redundant character comparisons. Any of several alternative pattern matching algorithms can be applied based on data set size or other factors. Finding a string in a text document via a function call is declarative, because the developer or user specifies a pattern to locate rather than how the computer should traverse the document, so that the calling code can avoid complex loops.

Full-text search indexing is a technique for making large volumes of text searchable quickly and accurately. Applying the technnique in imperative programming relies on data structures such as inverted indexes. Imperative code can use a hash table lookup to find index entries that match search terms and can use sharding, partitioning, and normalization techniques to handle large sets of terms. Creating and maintaining an inverted index can require significant computational workload and careful consideration of design tradeoffs.

Regular expressions can be used to extract specific information from text, based on specific patterns of interest. Software applications use them for programmatic pattern search, for input validation, and for extraction of information relevant to users. Though regular expressions allow for relatively complex pattern matching compared to a simple string search, such as a search using the Knuth-Morris-Pratt algorithm, the performance of a simple string search is significantly better than the performance of a regular expression search. Because regular expressions must be carefully designed based on a relatively sophisticated pattern format, code that uses simple string search is relatively easy to comprehend and maintain. Regular expressions applied to nested semi-structured data, such as JSON data, can be subject to subtle bugs. Furthermore, improperly designed regular expressions can be subject to catastrophic backtracking, a performance bug that occurs when a regular expression engine spends exponential time trying to evaluate a string that does not match a given pattern. Avoiding catastrophic backtracking requires careful attention to the regular expression pattern itself.

Delimiter search, or tokenization, is a technique for splitting text strings into segments or words using designated delimiters, such as punctuation marks or newline characters, as boundaries. One or more delimiters, such as commas, hyphens, or spaces, may be used in a single search through a text string. A greedy algorithm can be used to split a text string at every character present in a delimiter set.

Wildcard search is a technique used to search text for a pattern that contains wildcard characters, which can match zero or more characters in a text string. The Krauss wildcard-matching algorithm incorporated in declarative wildcard search functionality can be used, in the context of imperative programming, for semi-structured data extraction from text. The search provides a way for software to find a line, paragraph, or other portion of a text that contains a particular word or phrase. In a case-insensitive comparison that accepts a book title and selects its author's name from a sample of bibliographical citations, a wildcard search and delimiter search are coded with no loops, significantly outperforming simple string methods for a similar search.