---
title: "Load from Spark Dataframe"
component: "tigergraph-server"
version: "3.10"
module: "data-loading"
html_url: "/tigergraph-server/3.10/data-loading/load-from-spark-dataframe"
canonical_url: "https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe"
markdown_url: "https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md"
---

[View as HTML](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe)

# Load from Spark Dataframe

Table of Contents

* [Compatibility](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_compatibility)
* [Overview](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_overview)
* [Setup](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_setup)  
   * [Download the JARs](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_download_the_jars)  
   * [Create a Loading Job](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_create_a_loading_job)
* [Configure the Connector](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_configure_the_connector)  
   * [Configure Logging](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_configure_logging)
* [Use Case Examples](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_use_case_examples)  
   * [Authentication Methods](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_authentication_methods)  
   * [Batch Write Mode](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_batch_write_mode)  
   * [Streaming Mode with Spark Structured Streaming API](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_streaming_mode_with_spark_structured_streaming_api)  
   * [Load Data from Delta Lake](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_load_data_from_delta_lake)  
   * [Load Data from Iceberg](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_load_data_from_iceberg)  
   * [Load Data from Hudi](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_load_data_from_hudi)
* [Loading Statistics](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_loading_statistics)  
   * [Row Level Statistics](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_row_level_statistics)  
   * [Object Level Statistics](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_object_level_statistics)
* [Direct Upsert from Spark DataFrame](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_direct_upsert_from_spark_dataframe)  
   * [Upserting Vertices](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_upserting_vertices)  
   * [Upserting Edges](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_upserting_edges)  
   * [Attribute Mapping and Operations](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_attribute_mapping_and_operations)
* [Connector Options](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_connector_options)  
   * [Vertex Upsert Options](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_vertex_upsert_options)  
   * [Edge Upsert Options](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_edge_upsert_options)  
   * [Common Upsert Options](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_common_upsert_options)

The TigerGraph Spark Connector employs[Apache Spark](https://spark.apache.org) to read data from a Spark DataFrame (or Data Lake) and write to TigerGraph. This connector has multiple optimizations for high performance, scalability, and management.

## Compatibility

* TigerGraph 3.6.0 or higher. Job level loading statistics are only available for v3.10+.
* Spark 3.2 or higher with Scala 2.12 and Scala 2.13.
* JAVA 8 or higher.

Users can leverage it to connect TigerGraph to the Spark ecosystem and load data from any Spark-compatible data sources, such as:

* Distributed file system: HDFS, S3, GCS and ABS
* Streaming source: Kafka
* Data warehouse:[BigQuery](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-warehouse.md#_bigquery),[Snowflake](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-warehouse.md#_snowflake),[PostgreSql](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-warehouse.md#_postgresql), and Redshift
* Open table format:[Delta Lake](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_load_data_from_delta_lake),[Iceberg](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_load_data_from_iceberg)and [Hudi](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/load-from-spark-dataframe.md#_load_data_from_hudi)

> [!NOTE]
> The legacy [Spark Connection Via JDBC Driver](https://www.tigergraph.com/docs/tigergraph-server/3.10/data-loading/spark-connection-via-jdbc-driver.md) is deprecated. Please migrate to this new connector.

## Overview

The first step is to read data into a Spark dataframe. Then, using a TigerGraph loading job which maps data fields from the dataframe into graph elements, the connector pulls data from Spark into TigerGraph.

Spark dataframe example:

+--------------------+-------------+-------------+-----------+--------------------+------+
|        creationDate|           id|   locationIP|browserUsed|             content|length|
+--------------------+-------------+-------------+-----------+--------------------+------+
|2012-07-04T06:10:...|7696585588755| 46.23.82.182|    Firefox|                 LOL|     3|
|2012-08-22T17:22:...|8246341402699|  27.62.125.4|     Chrome|              roflol|     6|
|2012-05-08T21:02:...|7146829775042|  61.1.50.205|     Chrome|              roflol|     6|
|2012-11-22T01:25:...|9345853030654|190.95.68.192|    Firefox|About Sergei Eise...|    79|
|2012-11-11T08:59:...|9345853030710|166.75.225.76|     Chrome|                good|     4|
+--------------------+-------------+-------------+-----------+--------------------+------+

The connector concatenates columns to delimited data:

    - loading.separator = "|"
    - loading.eol = "\n"
the processed data would be:
    2012-07-04T06:10:43.489+00:00|7696585588755|46.23.82.182|Firefox|LOL|3
    2012-08-22T17:22:20.315+00:00|8246341402699|27.62.125.4|Chrome|roflol|6
    2012-05-08T21:02:39.145+00:00|7146829775042|61.1.50.205|Chrome|roflol|6
    2012-11-22T01:25:39.670+00:00|9345853030654|190.95.68.192|Firefox|About Sergei Eisenstein, pioneering SAbout Steven Spielberg, makers in thAbout|79
    2012-11-11T08:59:21.311+00:00|9345853030710|166.75.225.76|Chrome|good|4

The processed data will be sent to TigerGraph batch by batch:

    - loading.job = "load_Comment"
    - loading.filename = "file_Comment"

To let TigerGraph parse the data chunk correctly, please make sure you are setting `loading.separator` and `loading.eol` to characters which do not appear in your data fields.

## Setup

### Download the JARs

This connector can be downloaded from the Maven central repository for[Scala 2.12](https://central.sonatype.com/artifact/com.tigergraph/tigergraph-spark-connector%5F2.12/overview) and [Scala 2.13](https://central.sonatype.com/artifact/com.tigergraph/tigergraph-spark-connector%5F2.13/overview)The connector is available in 3 release formats:

* `tigergraph-spark-connector-<version>.jar`: The `JAR` file containing only the compiled classes of the connector, which does not include any dependencies.
* `tigergraph-spark-connector-<version>-jar-with-dependencies.jar`: The `JAR` file that includes compiled classes, as well as all the dependencies.
* `tigergraph-spark-connector-<version>.tar.gz`: The compressed `TAR` archive that includes `tigergraph-spark-connector-<version>.jar` and dependencies in separate `JAR` files.

To use the TigerGraph Spark connector in a Spark shell, use the `--jars` option:

spark-shell --jars tigergraph-spark-connector-<version>-jar-with-dependencies.jar

> [!NOTE]
> If you want to include the TigerGraph Spark Connector in your Spark installation, add the `JAR` with dependencies to Spark's `jars` folder.

### Create a Loading Job

This loading job should map the dataframe's columns to the desired graph element attributes.

Example of simple graph schema and loading job

gsql '
    CREATE VERTEX Comment (PRIMARY_ID id UINT, creationDate DATETIME, locationIP STRING, browserUsed STRING, content STRING, length UINT) WITH primary_id_as_attribute="TRUE", STATS="outdegree_by_edgetype"
    CREATE GRAPH demo_graph (*)
'

gsql -g demo_graph '
    CREATE LOADING JOB load_Comment FOR GRAPH test_graph {
        DEFINE FILENAME file_Comment;
        LOAD file_Comment
            TO VERTEX Comment VALUES ($1, $0, $2, $3, $4, $5) USING header="true", separator="|";
 }
'

## Configure the Connector

> [!NOTE]
> If you are using Spark Connector 0.2.0+, the following configuration parameters are required:`version`, `log.level`,`log.path`.

| Key | Default Value | Description | Group |
| --- | --- | --- | --- |
| `url` | (none) | The connection URL to TigerGraph cluster. It can be a list of URLs separated by comma for load balancing. Example:<http://192.168.1.1:14240>,<http://192.168.1.2:14240>,<http://192.168.1.3:14240> | General |
| `graph` | (none) | The graph name. |  |
| `username` | (none) | The GSQL username. | Authentication (You can choose any authentication method for data loading, but it's recommended to give username/password pair, which can generate and refresh token automatically.) |
| `password` | (none) | The GSQL password. | Authentication |
| `secret` | (none) | The GSQL secret. | Authentication |
| `token` | (none) | The Bearer token for RESTPP. | Authentication |
| `loading.job` | (none) | The GSQL loading job name. | Loading Job |
| `loading.filename` | (none) | The filename defined in the loading job. | Loading Job |
| `loading.separator` | , | The column separator. | Loading Job |
| `loading.eol` | \\n | The line separator. | Loading Job |
| `loading.batch.size.bytes` | 2097152 | The maximum batch size in bytes. | Loading Job |
| `loading.timeout.ms` | (none) | The loading timeout per batch. | Loading Job |
| `loading.max.percent.error` | (none) | The threshold of the error objects count. The loading job will be aborted when reaching the limit. Only available for TigerGraph version 3.10.0+. | Loading Job |
| `loading.max.num.error` | (none) | The threshold of the error objects percentage. The loading job will be aborted when reaching the limit. Only available for TigerGraph version 3.10.0+. | Loading Job |
| `loading.retry.interval.ms` | 5000 | The initial retry interval for transient server errors. | Loading Job |
| `loading.max.retry.attempts` | 10 | The maximum retry attempts for transient server errors. | Loading Job |
| `loading.max.retry.interval.ms` | 30000 | The maximum retry interval for transient server errors. | Loading Job |
| `ssl.mode` | basic | The SSL mode: basic, verifyCA and verifyHostname. When setting it to verifyCA and verifyHostname, the truststore file should be given. | SSL |
| `ssl.truststore` | (none) | Filename of the truststore which stores the SSL certificate chains. Add `--files /path/to/trust.jks` when submitting the Spark job. | SSL |
| `ssl.truststore.type` | JKS | Truststore type, e.g., JKS, PKCS12 | SSL |
| `ssl.truststore.password` | (none) | Password of the truststore. | SSL |
| `io.connect.timeout.ms` | 30000 | Connect timeout in ms. | Transport Timeout |
| `io.read.timeout.ms` | 60000 | Socket read timeout in ms. | Transport Timeout |
| `io.retry.interval.ms` | 5000 | The initial retry interval for transport timeout. | Transport Timeout |
| `io.max.retry.interval.ms` | 10000 | The maximum retry interval for transport timeout. | Transport Timeout |
| `io.max.retry.attempts` | 5 | The maximum retry attempts for transport timeout. | Transport Timeout |
| `oauth2.url` | (none) | The URL where the Client Credentials grant request should be sent for retrieving the access token. This URL is typically provided by the Identity Provider (IdP) such as Azure AD or Auth0.  For **Azure AD:** `<https://login.microsoftonline.com/{tenant%5Fid}/oauth2/v2.0/token>` For **Auth0:** `<https://{yourDomain}/oauth/token>` **Note:** Use the gadmin command to set the `Security.JWT.RSA.PublicKey` in advance for JWT verification. | Authentication |
| `oauth2.parameters` | (none) | A **stringified JSON** that carries parameters for retrieving the access token. The required parameters vary depending on the IdP.  For **Azure AD:** `{ "client_id": "{client_id}", "client_secret": "{client_secret}", "scope": "https://storage.azure.com/.default" }` For **Auth0:** `{ "client_id": "{client_id}", "client_secret": "{client_secret}", "audience": "{api_identifier}" }` | Authentication |

### Configure Logging

It's highly recommended to set log level to `info` to monitor the loading status. Configure the Spark log4j `conf/log4j2.properties` as follows, or other SLF4j bindings if any:

logger.tg.name = com.tigergraph
logger.tg.level = info

## Use Case Examples

**Some Helpful tips**

* **High Availabilty**

To ensure fault tolerance, provide a list of all node `URLs` in the `url` option for the connector:

```json
"url" -> "https://m1:14240,https://m2:14240,https://m3:14240,https://m4:14240",
```

### Authentication Methods

> [!NOTE]
> `OAuth2` authentication for Spark was originally added in **4.1.3** and backported to **3.10.4**. So, in version **3.10.4**, TigerGraph's Spark Connector supports `OAuth2` authentication, enabling it to request JWT tokens from third-party Identity Providers (IdPs) like `Azure AD` and `Auth0`, enhancing security and token management for Spark jobs.

We recommend using **OAuth2** for authentication. Our implementation supports automatic token requests and refresh, and is more secure that **username/password**, offering convenience and security for both batch and streaming jobs.

The Spark connector now supports the following authentication methods:

* `oauth2`: Recommended for both batch and streaming jobs. The token is automatically requested and refreshed using `oauth2.url` and `oauth2.parameters`.
* `token`: Suitable for one-time Spark jobs, but lacks support for refresh tokens.
* `username/password`: Works for both batch and streaming jobs, but is considered an older, less secure method. The token is automatically requested and refreshed.
* `secret`: Also works for batch and streaming jobs, and the token is managed using the GSQL secret, but is less secure compared to OAuth2.

### Batch Write Mode

Create the variables:

```scala
val GRAPH = "demo_graph"
val URL = "http(s)://hostname:port[,http(s)://hostname:port]*"
val USERNAME = "tigergraph"
val PASSWORD = "tigergraph"
val LOADING_JOB = "load_Comment"
val FILENAME = "file_Comment"
val SEPARATOR = "|"
val VERSION = "3.10.1"
val LOG_LEVEL = "2"
val LOG_PATH = "/tmp/tigergraph-spark-connector.log"
```

Read data from any Spark data sources into the dataframe:

```scala
val df = spark.read.json("path/to/person.json")
```

Batch write the data into TigerGraph:

```scala
df.write
    .format("tigergraph")
    .mode("append")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .save()
```

### Streaming Mode with Spark Structured Streaming API

Create the variables:

```scala
val GRAPH = "Social_Net"
val URL = "http(s)://hostname:port"
val USERNAME = "tigergraph"
val PASSWORD = "tigergraph"
val LOADING_JOB = "load_person"
val FILENAME = "f1"
val SEPARATOR = "|"
val VERSION = "3.10.1"
val LOG_LEVEL = "2"
val LOG_PATH = "/tmp/tigergraph-spark-connector.log"
```

Read data from any Spark streaming data sources into the dataframe:

```scala
val df = spark.readStream
    .format("kafka")
    .option("subscribe", "person")
    .load()
    .selectExpr("CAST(value AS STRING)").as[(String)]
```

Streaming write data to TigerGraph:

```scala
df.writeStream
    .outputMode("append")
    .format("tigergraph")
    .option("checkpointLocation", "/path/to/checkpoint")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .start()
    .awaitTermination()
```

### Load Data from Delta Lake

#### Batch Write

Load delta table to Spark dataframe:

```scala
val df = spark.read.format("delta")
    .load("/path/to/delta/table")
    .select(
        "creationDate",
        "id",
        "locationIP",
        "browserUsed",
        "content",
        "length"
    )
```

Batch write the data into TigerGraph:

```scala
df.write
    .format("tigergraph")
    .mode("append")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .save()
```

#### Streaming Write(CDC)

Streaming read from delta table:

```scala
val df = spark.readStream
    .format("delta")
    .option("readChangeFeed", "true")
    .load("/path/to/delta/table")
    .filter(
        $"_change_type" === "insert" || $"_change_type" === "update_postimage"
    )
    .select(
        "creationDate",
        "id",
        "locationIP",
        "browserUsed",
        "content",
        "length"
    )
```

Streaming write data to TigerGraph:

```scala
df.writeStream
    .outputMode("append")
    .format("tigergraph")
    .option("checkpointLocation", "/path/to/checkpoint")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .start()
    .awaitTermination()
```

For more details on Delta Lake see [Welcome to the Delta Lake documentation — Delta Lake Documentation](https://docs.delta.io/latest/index.html).

### Load Data from Iceberg

#### Batch Write

Load Iceberg table to Spark dataframe:

```scala
val df = spark.table("catalog.db.table")
    .select(
        "creationDate",
        "id",
        "locationIP",
        "browserUsed",
        "content",
        "length"
    )
```

Batch write the data into TigerGraph:

```scala
df.write
    .format("tigergraph")
    .mode("append")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .save()
```

#### Streaming Write(CDC)

Streaming read from Iceberg table:

```scala
val df = spark.readStream
    .format("iceberg")
    .option("stream-from-timestamp", 0L)
    .load("catalog.db.table")
    .select(
        "creationDate",
        "id",
        "locationIP",
        "browserUsed",
        "content",
        "length"
    )
```

Streaming write data to TigerGraph:

```scala
df.writeStream
    .outputMode("append")
    .format("tigergraph")
    .option("checkpointLocation", "/path/to/checkpoint")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .start()
    .awaitTermination()
```

For more details on Iceberg see [Iceberg Apache: Getting Started](https://iceberg.apache.org/docs/latest/spark-getting-started/)

### Load Data from Hudi

#### Batch Write

Load Hudi table to Spark dataframe:

```scala
val df = spark.read
    .format("hudi")
    .load("/path/to/hudi/table")
    .select(
        "creationDate",
        "id",
        "locationIP",
        "browserUsed",
        "content",
        "length"
    )
```

Batch write the data into TigerGraph

```scala
df.write
    .format("tigergraph")
    .mode("append")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .save()
```

#### Streaming Write(CDC)

Streaming read from Hudi table:

```scala
val df = spark.readStream
    .format("hudi")
    .load("/path/to/hudi/table")
    .select(
        "creationDate",
        "id",
        "locationIP",
        "browserUsed",
        "content",
        "length"
    )
```

Streaming write data to TigerGraph:

```scala
df.writeStream
    .outputMode("append")
    .format("tigergraph")
    .option("checkpointLocation", "/path/to/checkpoint")
    .options(
        Map(
            "graph" -> GRAPH,
            "url" -> URL,
            "username" -> USERNAME,
            "password" -> PASSWORD,
            "loading.job" -> LOADING_JOB,
            "loading.filename" -> FILENAME,
            "loading.separator" -> SEPARATOR,
            "version" -> VERSION,
            "log.level" -> LOG_LEVEL,
            "log.path" -> LOG_PATH
        )
    )
    .start()
    .awaitTermination()
```

For more details on Hudi see [Spark Guide | Apache Hudi](https://hudi.apache.org/docs/quick-start-guide/).

## Loading Statistics

When you configure the logging properly and set log level to `info`, the loading statistics will be logged.

There are 3 levels of stats:

* **Batch level**: data will be loaded to TigerGraph in micro batches. The counts of malformed or invalid data of the batch will be logged.
* **Partition level**: the data source can contain multiple partitions, and the log will show how many rows of the partition have been sent to TigerGraph.
* **Job Level (available for TigerGraph 3.10+)**: The overall loading statistics of the Spark job aggregated by TigerGraph service KAFKASTRM-LL. This requires providing username and password to the query/gsqlserver endpoint.

Sample loading statistics:

```scala
24/01/22 16:15:45 INFO TigerGraphBatchWrite: Overall loading statistics: [ {
    "overall" : {
        "duration" : 15792,
        "size" : 48675207,
        "progress" : 0,
        "startTime" : 1706770863875,
        "averageSpeed" : 29546,
        "id" : "test_graph.load_Comment.spark.all.1706770859889",
        "endTime" : 1706770879667,
        "currentSpeed" : 29546,
        "statistics" : {
            "fileLevel" : {
                "validLine" : 466594,
                "notEnoughToken" : 0,
                "tokenExceedsBuffer" : 0,
                "rejectLine" : 0
            },
            "objectLevel" : {
                "vertex" : [ {
                "validObject" : 466593,
                "typeName" : "Comment",
                "invalidPrimaryId" : 1
                } ]
            }
        }
    },
    "workers" : [ {
        "tasks" : [ {
            "filename" : "file_Comment"
        } ]
    }, {
    "tasks" : [ {
        "filename" : "file_Comment"
        } ]
    } ]
} ]
```

### Row Level Statistics

| Row Level Statistics | Description |
| --- | --- |
| `validLine` | Number of valid raw data lines parsed. |
| `rejectLine` | Number of raw data lines rejected by the reject line rule in the loading script. |
| `notEnoughToken` | Number of raw data lines with fewer tokens than what was specified by the loading script. |
| `badCharacter` | Number of raw data lines containing invalid characters. |
| `tokenExceedsBuffer` | Number of raw data lines containing oversize tokens (see `gadmin config get GSQL.OutputTokenBufferSize`). |
| `emptyLine` | Number of raw data lines that are empty. |

### Object Level Statistics

| Object Level Statistics | Description |
| --- | --- |
| `validObject` | Number of data records created. |
| `passedCondition` | Number of token lists which passed the WHERE predicate filter. |
| `failedCondition` | Number of token lists which failed the WHERE predicate filter. |
| `invalidPrimaryId` | Number of token lists where the id token is invalid. |
| `noIdFound` | Number of token lists where the id token is empty. |
| `invalidAttribute` | Number of token lists where at least one of the attribute tokens is invalid. |
| `incorrectFixedBinaryLength` | Number of token lists where at least one of the tokens corresponding to a UDT type attribute is invalid. |
| `invalidVertexType` | Number of token lists where at least one of the tokens corresponding to an edge type's source/target vertex type is invalid. |
| `tokenBankException` | Counts how many times a user-defined token function has thrown an exception during data loading. |

## Direct Upsert from Spark DataFrame

The Direct Upsert feature enables you to upsert data directly from a Spark DataFrame into vertices and edges in TigerGraph without the need to define a GSQL loading job. The connector sends data in batches to TigerGraph's REST++ upsert endpoint.

### Upserting Vertices

To upsert vertices, specify the vertex type and the field in the DataFrame corresponding to the vertex primary ID.

Upsert vertices of type `Person`

```scala
df.write
  .format("tigergraph")
  .mode("append")
  .option("graph", "my_graph")
  .option("upsert.vertex.type", "Person")
  .option("upsert.vertex.id.field", "person_id")
  .option("upsert.vertex.new_only", "true")
  .save()
```

### Upserting Edges

To upsert edges, specify the edge type, the source and target vertex types, and the fields in the DataFrame that correspond to the source and target vertex primary IDs.

Upsert edges of type `Follows`

```scala
df.write
  .format("tigergraph")
  .mode("append")
  .option("graph", "my_graph")
  .option("upsert.edge.type", "Follows")
  .option("upsert.edge.source.type", "User")
  .option("upsert.edge.source.id.field", "follower_id")
  .option("upsert.edge.target.type", "User")
  .option("upsert.edge.target.id.field", "followee_id")
  .option("upsert.edge.vertex_must_exist", "true")
  .save()
```

### Attribute Mapping and Operations

By default, the connector maps DataFrame column names to TigerGraph vertex or edge attribute names. Custom mappings and operations can be specified using `upsert.attribute.mapping` and `upsert.attribute.op`.

Upsert vertices with attribute mapping and operations

```scala
df.write
  .format("tigergraph")
  .mode("append")
  .option("graph", "my_graph")
  .option("upsert.vertex.type", "Person")
  .option("upsert.vertex.id.field", "pid")
  .option("upsert.attribute.mapping", "name:fullName, years:age, scoreDelta:score")
  .option("upsert.attribute.op", "score:+, age:max")
  .save()
```

## Connector Options

The direct upsert feature is configured through options passed to the Spark DataFrame writer.

### Vertex Upsert Options

| Key | Required | Default Value | Description |
| --- | --- | --- | --- |
| `upsert.vertex.type` | Yes | (none) | The target vertex type. |
| `upsert.vertex.id.field` | Yes | (none) | The field in the DataFrame to be used as the vertex ID. |
| `upsert.vertex.new_only` | No | false | If true, only insert new vertices (no updates). |
| `upsert.vertex.update_only` | No | false | If true, only update existing vertices (no inserts). |

### Edge Upsert Options

| Key | Required | Default Value | Description |
| --- | --- | --- | --- |
| `upsert.edge.type` | Yes | (none) | The edge type to upsert into. |
| `upsert.edge.source.type` | Yes | (none) | The source vertex type. |
| `upsert.edge.source.id.field` | Yes | (none) | The field for the source vertex ID. |
| `upsert.edge.target.type` | Yes | (none) | The target vertex type. |
| `upsert.edge.target.id.field` | Yes | (none) | The field for the target vertex ID. |
| `upsert.edge.vertex_must_exist` | No | false | If true, insert the edge only if both vertices exist. |
| `upsert.edge.source_vertex_must_exist` | No | false | Require only the source vertex to exist. |
| `upsert.edge.target_vertex_must_exist` | No | false | Require only the target vertex to exist. |

### Common Upsert Options

| Key | Required | Default Value | Description |
| --- | --- | --- | --- |
| `upsert.attribute.mapping` | No | None | A comma-separated string that defines how DataFrame field names map to TigerGraph attribute names. Each entry follows the format `field_name:attribute_name`. If not set, attribute names must exactly match the DataFrame field names. |
| `upsert.attribute.op` | No | \= | A comma-separated string specifying the update operation for each target attribute. Each entry follows the format `attribute_name:op`. If not specified, attributes default to `=` (overwrite). |
| `upsert.ack` | No | all | `"all"` (wait for GPEs) or `"none"` (fire and forget). |
| `upsert.atomic` | No | false | If true, make each request atomic. |
| `upsert.batch.size.in.rows` | No | 1000 | Maximum number of rows per batch request. |
| `upsert.timeout.ms` | No | 0 | Timeout per request in milliseconds. A value of 0 means no timeout. |
| `upsert.retry.interval.ms` | No | 5000 | Initial retry interval after failure. |
| `upsert.max.retry.interval.ms` | No | 300000 | Maximum retry interval after backoff. |
| `upsert.max.retry.attempts` | No | 10 | Maximum number of retry attempts. |
