T
- stream tuple typepublic class FileWriterPolicy<T>
extends java.lang.Object
implements org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
IFileWriterPolicy
implementation.
The policy implements strategies for:
FileWriterPolicy
can be
created.
Sample use:
FileWriterPolicy<String> policy = new FileWriterPolicy( FileWriterFlushConfig.newImplicitConfig(), FileWriterCycleConfig.newCountBasedConfig(1000), FileWriterRetentionConfig.newCountBasedConfig(10)); String basePathname = "/some/directory/and_base_name"; TStream<String> streamToWrite = ... FileStreams.textFileWriter(streamToWrite, () -> basePathname, () -> policy)
Constructor and Description |
---|
FileWriterPolicy()
Create a new file writer policy instance.
|
FileWriterPolicy(FileWriterFlushConfig<T> flushConfig,
FileWriterCycleConfig<T> cycleConfig,
FileWriterRetentionConfig retentionConfig)
Create a new file writer policy instance.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
java.nio.file.Path |
closeActiveFile(java.nio.file.Path path) |
FileWriterCycleConfig<T> |
getCycleConfig()
Get the policy's active file cycle configuration
|
FileWriterFlushConfig<T> |
getFlushConfig()
Get the policy's active file flush configuration
|
java.nio.file.Path |
getNextActiveFilePath() |
FileWriterRetentionConfig |
getRetentionConfig()
Get the policy's retention configuration
|
protected java.nio.file.Path |
hookGenerateFinalFilePath(java.nio.file.Path path)
Generate the final file path for the active file.
|
protected java.nio.file.Path |
hookGenerateNextActiveFilePath()
Generate the path for the next active file.
|
protected void |
hookRenameFile(java.nio.file.Path activePath,
java.nio.file.Path finalPath)
"Rename" the active file to the final path.
|
void |
initialize(java.lang.String basePathname,
java.io.Flushable flushable,
java.io.Closeable closeable) |
boolean |
shouldCycle() |
boolean |
shouldFlush() |
java.lang.String |
toString() |
void |
wrote(T tuple,
long nbytes) |
public FileWriterPolicy()
The configuration is:
FileWriterPolicy(FileWriterFlushConfig, FileWriterCycleConfig, FileWriterRetentionConfig)
public FileWriterPolicy(FileWriterFlushConfig<T> flushConfig, FileWriterCycleConfig<T> cycleConfig, FileWriterRetentionConfig retentionConfig)
flushConfig
, cycleConfig
and retentionConfig
specify the configuration of the various controls.
The active file and final file pathnames are based
on the basePathname
received in
initialize(String, Flushable, Closeable)
.
Where parent
and baseLeafname
are the
parent path and file name respectively of basePathname
:
parent/.baseLeafname
"parent/baseLeafname_YYYYMMDD_HHMMSS[_<n>]
where the optional _<n>
suffix is only present if needed
to distinguish a file from the previously finalized file.
<n>
starts at 1 and is monotonically incremented.
flushConfig
- active file flush control configurationcycleConfig
- active file cycle control configurationretentionConfig
- final file retention control configurationpublic void close()
close
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
public FileWriterFlushConfig<T> getFlushConfig()
public FileWriterCycleConfig<T> getCycleConfig()
public FileWriterRetentionConfig getRetentionConfig()
public void initialize(java.lang.String basePathname, java.io.Flushable flushable, java.io.Closeable closeable)
initialize
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
public void wrote(T tuple, long nbytes)
wrote
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
public boolean shouldFlush()
shouldFlush
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
public boolean shouldCycle()
shouldCycle
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
public java.nio.file.Path getNextActiveFilePath()
getNextActiveFilePath
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
public java.nio.file.Path closeActiveFile(java.nio.file.Path path) throws java.io.IOException
closeActiveFile
in interface org.apache.edgent.connectors.file.runtime.IFileWriterPolicy<T>
java.io.IOException
protected java.nio.file.Path hookGenerateFinalFilePath(java.nio.file.Path path)
The default implementation yields:
final file names are basePathname_YYYYMMDD_HHMMSS[_<n>]
where the optional _<n>
suffix is only present if needed
to distinguish a file from the previously finalized file.
<n>
starts at 1 and is monitonically incremented.
This hook method can be overridden.
Note, the implementation must handle the unlikely, but happens in tests, case where files are cycling very fast (multiple per sec) and the retention config tosses some within that same second. I.e., avoid generating final path sequences like:
leaf_YYYYMMDD_103099 leaf_YYYYMMDD_103099_1 leaf_YYYYMMDD_103099_2 delete leaf_YYYYMMDD_103099 -- retention cnt was 2 leaf_YYYYMMDD_103099 // should be _3
path
- the active file path to finalizeprotected java.nio.file.Path hookGenerateNextActiveFilePath()
The default implementation yields parent/.baseLeafname
from basePathname
.
This hook method can be overridden.
See IFileWriterPolicy.getNextActiveFilePath()
regarding
constraints.
protected void hookRenameFile(java.nio.file.Path activePath, java.nio.file.Path finalPath) throws java.io.IOException
The default implementation uses java.io.File.renameTo()
and works for the default hookGenerateNextActiveFilePath()
and hookGenerateFinalFilePath(Path path)
implementations.
This hook method can be overridden.
activePath
- path of the active filefinalPath
- path to the final destinationjava.io.IOException
- on failurepublic java.lang.String toString()
toString
in class java.lang.Object
Copyright © 2016 The Apache Software Foundation. All Rights Reserved - bbe71fa-20161201-1641