Packages

trait OutputFacade extends ServiceFacadeWithDatabase with ServiceFacade[OutputFacade]

The interface for the external API of the Output Service.

The Output Service is responsible for the management of output data and metadata from runs of sources, pipelines and batches. It creates point-in-time snapshots of the configuration which are used to maintain a complete revision history knowing exactly which other data sets a given data set depends on, and which version of the configuration was used to build them.

Linear Supertypes
ServiceFacade[OutputFacade], ServiceFacadeWithDatabase, ServiceFacadeBase, AnyRef, Any
Known Subclasses
OutputWebFacade
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OutputFacade
  2. ServiceFacade
  3. ServiceFacadeWithDatabase
  4. ServiceFacadeBase
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract val environment: String
    Definition Classes
    ServiceFacadeBase
  2. abstract def getArtifactContentForSnapshotForPipeline(pipelineId: String, pipelineSnapshotVersion: UUID, transformId: String, name: String, pipelineTag: String, outputStream: OutputStream): Unit

    Gets the content for a specific artifact in a given pipeline snapshot.

    Gets the content for a specific artifact in a given pipeline snapshot.

    pipelineId

    The unique id of the pipeline.

    pipelineSnapshotVersion

    The UUID of the snapshot.

    transformId

    The unique id of the transform in the pipeline.

    name

    The name of the artifact.

    pipelineTag

    The tag of the snapshot.

    outputStream

    The stream where data is written to.

  3. abstract def getArtifactsForSnapshotForPipeline(id: String, snapshotId: UUID): Array[ArtifactsInfo]

    Gets the artifact metadata associated a specific pipeline and snapshot.

    Gets the artifact metadata associated a specific pipeline and snapshot.

    id

    The unique id of the pipeline.

    snapshotId

    The UUID of the snapshot.

  4. abstract def getCacheStatistics: Array[CacheStatistics]
    Definition Classes
    ServiceFacadeBase
  5. abstract def getDatabase(outputStream: OutputStream): Unit
    Definition Classes
    ServiceFacadeWithDatabase
  6. abstract def getDatabasePath: String
    Definition Classes
    ServiceFacadeWithDatabase
  7. abstract def getFile(path: String, outputStream: OutputStream): Unit
    Definition Classes
    ServiceFacadeBase
  8. abstract def getLatestForSnapshotsForBatch(id: String, tag: String, date: LocalDateTime): BatchSnapshotInfo

    Gets the most recent snapshot relative to a given date for a specific batch and tag.

    Gets the most recent snapshot relative to a given date for a specific batch and tag.

    id

    The unique id of the batch.

    tag

    The tag of the snapshot.

    date

    The date to use as the upper bound. LocalDateTime.MAX not supported. Use DateTimeSupport.maxDate instead.

  9. abstract def getLatestForSnapshotsForPipeline(id: String, tag: String, date: LocalDateTime): PipelineSnapshotInfo

    Gets the most recent snapshot relative to a given date for a specific pipeline and tag.

    Gets the most recent snapshot relative to a given date for a specific pipeline and tag.

    id

    The unique id of the pipeline.

    tag

    The tag of the snapshot.

    date

    The date to use as the upper bound. LocalDateTime.MAX not supported. Use DateTimeSupport.maxDate instead.

  10. abstract def getLatestForSnapshotsForSource(id: String, tag: String, date: LocalDateTime): SourceSnapshotInfo

    Gets the most recent snapshot relative to a given date for a specific source and tag.

    Gets the most recent snapshot relative to a given date for a specific source and tag.

    id

    The unique id of the source.

    tag

    The tag of the snapshot.

    date

    The date to use as the upper bound. LocalDateTime.MAX not supported. Use DateTimeSupport.maxDate instead.

  11. abstract def getLogLevel: String
    Definition Classes
    ServiceFacadeBase
  12. abstract def getOutputForSnapshotsForPipeline(id: String, snapshotIds: Array[UUID], transformId: String, outputStream: OutputStream, locale: String = getOutputDefaultLocale, limit: Option[Int] = None, maxDecimals: Int = getOutputDefaultMaxDecimals, serializeAsJson: Boolean = false): Unit

    Gets the output data set for a specific pipeline transform and specific snapshots.

    Gets the output data set for a specific pipeline transform and specific snapshots. Data is written incrementally, but this may still take a long time to complete for large data sets.

    id

    The unique id of the pipeline.

    snapshotIds

    The UUID values of the snapshots.

    transformId

    The unique id of the transform in the pipeline.

    outputStream

    The stream where data is written to.

    locale

    The locale to use when formatting numbers, i.e. 'da'.

    limit

    A limit on the number of data rows to write.

    maxDecimals

    The maximum number of decimals to include when formatting numbers.

    serializeAsJson

    Serialize the output as JSON array of arrays or as a csv with a ';' as the separator in either case the first row/line is the header.

  13. abstract def getOutputForSnapshotsForSource(id: String, snapshotIds: Array[UUID], outputStream: OutputStream, locale: String = getOutputDefaultLocale, limit: Option[Int] = None, maxDecimals: Int = getOutputDefaultMaxDecimals, serializeAsJson: Boolean = false): Unit

    Gets the output data set for a specific source and specific snapshots.

    Gets the output data set for a specific source and specific snapshots. Data is written incrementally, but this may still take a long time to complete for large data sets.

    id

    The unique id of the source.

    snapshotIds

    The UUID values of the snapshots.

    outputStream

    The stream data is written to.

    locale

    The locale to use when formatting numbers, i.e. 'da'.

    limit

    A limit on the number of data rows to write.

    maxDecimals

    The maximum number of decimals to include when formatting numbers.

    serializeAsJson

    Serialize the output as JSON array of arrays or as a csv with a ';' as the separator in either case the first row/line is the header.

  14. abstract def getPerformanceMetrics: PerformanceMetrics
    Definition Classes
    ServiceFacadeBase
  15. abstract def getSnapshotForBatch(id: String, tag: String, date: LocalDateTime): BatchSnapshotInfo

    Gets a specific snapshot for a specific batch.

    Gets a specific snapshot for a specific batch. Batch snapshots are uniquely determined by their batch id, tag and date.

    id

    The unique id of the batch.

    tag

    The tag of the snapshot.

    date

    The date of the snapshot.

  16. abstract def getSnapshotForPipeline(id: String, snapshotId: UUID): PipelineSnapshotInfo

    Gets a specific snapshot for a specific pipeline.

    Gets a specific snapshot for a specific pipeline.

    id

    The unique id of the pipeline.

    snapshotId

    The UUID of the snapshot.

  17. abstract def getSnapshotForSource(id: String, snapshotId: UUID): SourceSnapshotInfo

    Gets a specific snapshot for a specific source.

    Gets a specific snapshot for a specific source.

    id

    The unique id of the source.

    snapshotId

    The UUID of the snapshot.

  18. abstract def getSnapshotsForBatch(id: String, includeVolatile: Boolean = false, snapshotIds: Option[Array[LocalDateTime]] = None): Array[BatchSnapshotInfo]

    Gets all snapshots for a specific batch filtered on given parameters.

    Gets all snapshots for a specific batch filtered on given parameters.

    id

    The unique id of the batch.

    includeVolatile

    Whether to include snapshots with an expiration date.

    snapshotIds

    An optional set of specific snapshot ids to use as filter.

  19. abstract def getSnapshotsForBatchForTag(id: String, tag: String, includeVolatile: Boolean = false, snapshotIds: Option[Array[LocalDateTime]] = None): Array[BatchSnapshotInfo]

    Gets all snapshots for a specific batch and tag filtered on given parameters.

    Gets all snapshots for a specific batch and tag filtered on given parameters.

    id

    The unique id of the batch.

    tag

    A tag to require on the snapshot.

    includeVolatile

    Whether to include snapshots with an expiration date.

    snapshotIds

    An optional set of specific snapshot ids to use as filter.

  20. abstract def getSnapshotsForPipeline(id: String, includeDeleted: Boolean = false, includeVolatile: Boolean = false, snapshotIds: Option[Array[UUID]] = None): Array[PipelineSnapshotInfo]

    Gets all snapshots for a specific pipeline filtered on given parameters.

    Gets all snapshots for a specific pipeline filtered on given parameters.

    id

    The unique id of the pipeline.

    includeVolatile

    Whether to include snapshots with an expiration date.

    snapshotIds

    An optional set of specific snapshot ids to use as filter.

  21. abstract def getSnapshotsForPipelineForTag(id: String, tag: String, includeDeleted: Boolean = false, includeVolatile: Boolean = false, snapshotIds: Option[Array[UUID]] = None): Array[PipelineSnapshotInfo]

    Gets all snapshots for a specific pipeline and tag filtered on given parameters.

    Gets all snapshots for a specific pipeline and tag filtered on given parameters.

    id

    The unique id of the pipeline.

    tag

    A tag to require on the snapshot.

    includeDeleted

    Whether to include snapshots that have either been deleted or are not yet computed.

    includeVolatile

    Whether to include snapshots with an expiration date.

    snapshotIds

    An optional set of specific snapshot ids to use as filter.

  22. abstract def getSnapshotsForSource(id: String, includeDeleted: Boolean = false, includeVolatile: Boolean = false, snapshotIds: Option[Array[UUID]] = None): Array[SourceSnapshotInfo]

    Gets all snapshots for a specific source filtered on given parameters.

    Gets all snapshots for a specific source filtered on given parameters.

    id

    The unique id of the source.

    includeVolatile

    Whether to include snapshots with an expiration date.

    snapshotIds

    An optional set of specific snapshot ids to use as filter.

  23. abstract def getSnapshotsForSourceForTag(id: String, tag: String, includeDeleted: Boolean = false, includeVolatile: Boolean = false, snapshotIds: Option[Array[UUID]] = None): Array[SourceSnapshotInfo]

    Gets all snapshots for a specific source and tag filtered on given parameters.

    Gets all snapshots for a specific source and tag filtered on given parameters.

    id

    The unique id of the source.

    tag

    A tag to require on the snapshot.

    includeDeleted

    Whether to include snapshots that have either been deleted or are not yet computed.

    includeVolatile

    Whether to include snapshots with an expiration date.

    snapshotIds

    An optional set of specific snapshot ids to use as filter.

  24. abstract def getTableMetadata(tableName: String): TableMetadataInfo

    Gets the metadata associated with a given data set.

    Gets the metadata associated with a given data set. For internal use only.

    tableName

    The name of the data set generated from OutputAccess.id.

  25. abstract def getVersion: String
    Definition Classes
    ServiceFacadeBase
  26. abstract def isReady: Boolean
    Definition Classes
    ServiceFacadeBase
  27. abstract def postCachePurge(id: Option[String]): Unit
    Definition Classes
    ServiceFacadeBase
  28. abstract def postCheckpointForSnapshotForPipeline(id: String, snapshotId: UUID, checkpoint: SnapshotCheckpoint): Unit

    Appends a checkpoint to a specific pipeline snapshot.

    Appends a checkpoint to a specific pipeline snapshot. Checkpoints are used to allow partial reuse of data in a previously created snapshot. This way a transient failure would not necessarily require running all transforms in the pipeline again. For internal use only.

    id

    The unique id of the pipeline.

    snapshotId

    The UUID of the snapshot.

    checkpoint

    The checkpoint information to append.

  29. abstract def postCheckpointForSnapshotForSource(id: String, snapshotId: UUID, checkpoint: SnapshotCheckpoint): Unit

    Appends a checkpoint to a specific source snapshot.

    Appends a checkpoint to a specific source snapshot. Checkpoints are used to allow partial reuse of data in a previously created snapshot. This way a transient failure would not necessarily require running all transforms in the pipeline again. For internal use only.

    id

    The unique id of the source.

    snapshotId

    The UUID of the snapshot.

    checkpoint

    The checkpoint information to append.

  30. abstract def postDatabaseBackup(encryptionEnabled: Option[Boolean], encryptionMasterKeyIdentifier: Option[String], outputRepository: String = EnvironmentVariables.output_repository): Unit
    Definition Classes
    ServiceFacadeWithDatabase
  31. abstract def postDeleteColumnsForSnapshotForPipeline(id: String, snapshotId: UUID, transform: String, columns: List[String]): Unit

    Attempts to restore a deleted pipeline snapshot to a persisted state.

    Attempts to restore a deleted pipeline snapshot to a persisted state. The restore is dependent on versioning in the underlying data store.

    id

    The unique id of the pipeline.

    snapshotId

    The UUID value of the snapshot.

    transform

    The name of the transform to delete from.

    columns

    The names of the columns to delete snapshot.

  32. abstract def postDeleteColumnsForSnapshotForSource(id: String, snapshotId: UUID, columns: List[String]): Unit

    Deletes one or more columns from a persisted source snapshot.

    Deletes one or more columns from a persisted source snapshot.

    id

    The unique id of the source.

    snapshotId

    The UUID value of the snapshot.

    columns

    The names of the columns to delete snapshot.

  33. abstract def postDeleteForSnapshotForBatch(id: String, tag: String, dates: LocalDateTime*): Unit

    Deletes a list of specific batch snapshots and marks all included source and pipeline snapshots, that are not used by other batch snapshots, as deleted.

    Deletes a list of specific batch snapshots and marks all included source and pipeline snapshots, that are not used by other batch snapshots, as deleted.

    id

    The unique id of the batch.

    tag

    The tag of the snapshot.

    dates

    The dates for the snapshots.

  34. abstract def postDeleteForSnapshotForPipeline(id: String, snapshotIds: UUID*): Unit

    Marks a list of snapshots for a specific pipeline as deleted.

    Marks a list of snapshots for a specific pipeline as deleted. The underlying data will eventually be purged dependent on EnvironmentVariables.DATASOLVR_OUTPUT_PURGE_AFTER_DAYS.

    id

    The unique id of the pipeline.

    snapshotIds

    The UUID values for the snapshots.

  35. abstract def postDeleteForSnapshotForSource(id: String, snapshotIds: UUID*): Unit

    Marks a list of snapshots for a specific source as deleted.

    Marks a list of snapshots for a specific source as deleted. The underlying data will eventually be purged dependent on EnvironmentVariables.DATASOLVR_OUTPUT_PURGE_AFTER_DAYS.

    id

    The unique id of the source.

    snapshotIds

    The UUID values for the snapshots.

  36. abstract def postGenerateForSnapshotForBatch(batch: BatchContent, tag: String, date: LocalDateTime = postSnapshotDefaultDate()): BatchSnapshotInfo

    Generates a snapshot for a given batch definition, tag and date.

    Generates a snapshot for a given batch definition, tag and date. This snapshot is not stored in the database and cannot be referenced.

    batch

    The batch definition.

    tag

    The tag to use when creating the snapshot.

    date

    The date to associate with the snapshot.

  37. abstract def postGenerateForSnapshotForPipeline(pipeline: PipelineContent, date: LocalDateTime = postSnapshotDefaultDate(), sourceSnapshots: List[SourceSnapshotInfo] = List(), pipelineSnapshots: List[PipelineSnapshotInfo] = List()): PipelineSnapshotInfo

    Generates a snapshot for a given pipeline definition and date.

    Generates a snapshot for a given pipeline definition and date. This snapshot is not stored in the database and cannot be referenced.

    pipeline

    The pipeline definition.

    date

    The date to associate with the snapshot.

    sourceSnapshots

    A collection of generated source snapshots that can be used when building the snapshot.

    pipelineSnapshots

    A collection of generated pipeline snapshots that can be used when building the snapshot.

  38. abstract def postGenerateForSnapshotForSource(source: SourceContent, date: LocalDateTime = postSnapshotDefaultDate()): SourceSnapshotInfo

    Generates a snapshot for a given source definition and date.

    Generates a snapshot for a given source definition and date. This snapshot is not stored in the database and cannot be referenced.

    source

    The source definition.

    date

    The date to associate with the snapshot.

  39. abstract def postIsPersistedForSnapshotForPipeline(id: String, snapshotIds: Array[UUID], isPersisted: Boolean): Unit

    Update the persistence status of a collection of snapshots for a specific pipeline.

    Update the persistence status of a collection of snapshots for a specific pipeline. For internal use only.

    id

    The unique id of the pipeline.

    snapshotIds

    The UUID values of the snapshots.

    isPersisted

    The persistence value to set.

  40. abstract def postIsPersistedForSnapshotForSource(id: String, snapshotIds: Array[UUID], isPersisted: Boolean): Unit

    Update the persistence status of a collection of snapshots for a specific source.

    Update the persistence status of a collection of snapshots for a specific source. For internal use only.

    id

    The unique id of the source.

    snapshotIds

    The UUID values of the snapshots.

    isPersisted

    The persistence value to set.

  41. abstract def postRestart: Unit
    Definition Classes
    ServiceFacadeBase
  42. abstract def postRestoreForSnapshotForPipeline(id: String, snapshotId: UUID): Unit

    Attempts to restore a deleted pipeline snapshot to a persisted state.

    Attempts to restore a deleted pipeline snapshot to a persisted state. The restore is dependent on versioning in the underlying data store.

    id

    The unique id of the pipeline.

    snapshotId

    The UUID value of the snapshot.

  43. abstract def postRestoreForSnapshotForSource(id: String, snapshotId: UUID): Unit

    Attempts to restore a deleted source snapshot to a persisted state.

    Attempts to restore a deleted source snapshot to a persisted state. The restore is dependent on versioning in the underlying data store.

    id

    The unique id of the source.

    snapshotId

    The UUID value of the snapshot.

  44. abstract def postSnapshotForBatch(id: String, tag: String, date: LocalDateTime = postSnapshotDefaultDate(), expires: Option[LocalDateTime] = None, updates: BatchUpdates = BatchUpdates(List(), List())): BatchSnapshotInfo

    Creates a snapshot for a specific batch for a given tag and date.

    Creates a snapshot for a specific batch for a given tag and date. This creates snapshots for all sources and pipelines with results missing from the batch, with a requested update or with a dependency on a requested update. Untagged pipelines and sources may be used if no matching tag exists.

    id

    The unique id of the batch.

    tag

    The tag to use when creating the snapshot.

    date

    The date to associate with the snapshot.

    expires

    An optional expiration time for the snapshot.

    updates

    A specification of source and pipelines that should be updated. This will also update dependencies in the batch.

  45. abstract def postSnapshotForPipeline(id: String, tag: String, date: LocalDateTime = postSnapshotDefaultDate(), allowOverwrite: Boolean = false): PipelineSnapshotInfo

    Creates a snapshot for a specific pipeline for a given tag and date.

    Creates a snapshot for a specific pipeline for a given tag and date. Untagged pipelines and sources may be used if no matching tag exists.

    id

    The unique id of the pipeline.

    tag

    The tag to use when creating the snapshot

    date

    The date to associate with the snapshot.

    allowOverwrite

    Whether to allow deletion of an existing snapshot with the same date and tag, and no dependent snapshots.

  46. abstract def postSnapshotForSource(id: String, tag: String, date: LocalDateTime = postSnapshotDefaultDate(), allowOverwrite: Boolean = false): SourceSnapshotInfo

    Creates a snapshot for a specific source for a given tag and date.

    Creates a snapshot for a specific source for a given tag and date. Untagged sources and files may be used if no matching tag exists.

    id

    The unique id of the source.

    tag

    The tag to use when creating the snapshot

    date

    The date to associate with the snapshot.

    allowOverwrite

    Whether to allow deletion of an existing snapshot with the same date and tag, and no dependent snapshots.

  47. abstract def postTableMetadata(metadata: TableMetadataInfo): Unit

    Updates the collection of data set metadata.

    Updates the collection of data set metadata. For internal use only.

    metadata

    The metadata to insert or update.

  48. abstract def postVacuumDatabase: Unit
    Definition Classes
    ServiceFacadeWithDatabase
  49. abstract def setLogLevel(level: String): Unit
    Definition Classes
    ServiceFacadeBase
  50. abstract def withTimeout(seconds: Int): OutputFacade
    Definition Classes
    ServiceFacade
  51. abstract def withToken(token: UUID): OutputFacade
    Definition Classes
    ServiceFacade

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  9. val getOutputDefaultLocale: String

    The default value for locale used in getOutputForSnapshotsForSource, getOutputForSnapshotsForPipeline.

    The default value for locale used in getOutputForSnapshotsForSource, getOutputForSnapshotsForPipeline. The value is equal to EnvironmentVariables.DATASOLVR_DEFAULT_OUTPUT_LOCALE.

  10. val getOutputDefaultMaxDecimals: Int

    The default value for maxDecimals used in getOutputForSnapshotsForSource, getOutputForSnapshotsForPipeline.

    The default value for maxDecimals used in getOutputForSnapshotsForSource, getOutputForSnapshotsForPipeline. The value is equal to EnvironmentVariables.DATASOLVR_DEFAULT_MAX_DECIMALS.

  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  16. val postSnapshotDefaultDate: () => LocalDateTime

    The default value for date used in postSnapshotForSource, postSnapshotForPipeline and postSnapshotForBatch.

    The default value for date used in postSnapshotForSource, postSnapshotForPipeline and postSnapshotForBatch. The value is a function that returns the current UTC time.

  17. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. def waitIsReady(timeoutSeconds: Int, pollIntervalSeconds: Int): Unit
    Definition Classes
    ServiceFacadeBase

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from ServiceFacade[OutputFacade]

Inherited from ServiceFacadeWithDatabase

Inherited from ServiceFacadeBase

Inherited from AnyRef

Inherited from Any

Ungrouped