All Collections
Data Sources
Batch
Data Pipeline Batch File Delivery Tips and Best Practices
Data Pipeline Batch File Delivery Tips and Best Practices

How to connect and transfer files to your batch data pipeline

Openbridge Support avatar
Written by Openbridge Support
Updated over a week ago

How To Deliver Data

Transfer Protocols

Pipeline supports transfer methods: SFTP 

FTPES and FTP protocols are not supported. FTP is an insecure protocol that should only be used in limited cases or on networks you trust.

SFTP is integrated into many graphical tools or can be done directly via the command line.

Secure File Transfer Protocol (SFTP)

SFTP is based on the SSH2 protocol, which encodes activity over a secure channel. Unlike FTP, SSH2 only uses a single TCP connection and sends multiple transfers, or "channels", over that single connection.

Openbridge currently supports public key and password authentication for both SFTP and SCP file transfer protocols. If you need to use public key authentication, please submit a support ticket to support@openbridge.com and we can set that up for you. We do NOT support any shell access. Also, not all SFTP commands are accepted.

Connection Details:

    Host:               pipeline-01.openbridge.io
    Port:               22 or 443
    Protocol:           SFTP
    User:               Provided separately
    Password:           Provided separately


NOTE: If you need a  host with a static IP address, you can use the host name of static.pipeline.openbridge.io .

File Transfer Protocol (FTP)

Openbridge does support the use of FTP. We recognize that there are some systems that can only deliver data via FTP. For example, many of the Adobe export processes typically occur via FTP. However, it should be noted that the use of FTP offers no encryption for connection and data transport. Using the FTP protocol is regarded to be unsafe. It is therefore advisable to use SFTP connections to ensure that data is securely transferred.

If you need FTP access, please contact support.

Check Your Firewall

If you are having connection difficulties, please make sure you can make outbound network connections for SFTP make sure outbound port:22 or port:443 is open. Either can be used to connect.

Blocked Files

To help protect the integrity of the data sent to Openbridge, we do not allow delivery of certain files. For example, you are now allowed to send a file called app.exe or my.sql. This reduces the potential for introducing unwanted or malicious software threats. The following is a sample of the blocked files:

    ade|adp|app|ai|asa|ashx|asmx|asp|bas|bat|cdx|cer|cgi|chm|class|cmd|com|config|cpl|crt|csh|dmg|doc|docx|dll|eps|exe|fxp|ftaccess|hlp|hta|htr|htaccess|htw|html|htm|ida|idc|idq|ins|isp|its|jse|ksh|lnk|mad|maf|mag|mam|maq|mar|mas|mat|mau|mav|maw|mda|mdb|mde|mdt|mdw|mdz|msc|msh|msh1|msh1xml|msh2|msh2xml|mshxml|msi|msp|mst|ops|pdf|php|php3|php4|php5|pcd|pif|prf|prg|printer|pst|psd|rar|reg|rem|scf|scr|sct|shb|shs|shtm|shtml|soap|stm|tgz|taz|url|vb|vbe|vbs|ws|wsc|wsf|wsh|xls|xlsx|xvd

If you attempt to send a file that matches a blocked file then the transfer will not be allowed.

Do Not Process Files

Normally data delivered to the SFTP server will be routed to the target data warehouse. However, there is one exception. Any files placed into a testing directory will be ignored. By default each user has a /testing folder generated for them upon login. This is a safe place to upload files without the risk of the data flowing down stream to the data warehouse.

Hidden Files

Hidden files are not allowed on the server. Hidden files have a dot prefix . in the file name. For example, .file.txt or .foo would be considered hidden files and be blocked.

The following are examples of names using a dot prefix:

    .file.txt
    .file.csv
    .file
    ..file

If you attempt to send a file that contains a . prefix the transfer will not be allowed.

Error Handling

In most cases, file transfers occur without incident. However, there are situations where an error may arise. Troubleshooting an error can be a difficult and time consuming endeavor, especially in cases where a system may be sending 1000s of files a day.

Bulk Transfers

Openbridge does not have visibility into the what should be sent from a source system. It only knows what was sent by the source system. For example, if there are 100 files in a source system and only 50 were sent, Openbridge is only aware of the 50 delivered files. We will not know that an additional 50 files were not delivered.

Manifests

The source system should be tracking what was delivered and what was not delivered. We suggest that a manifest of files is maintained in the source system. This manifest would identify the files to be delivered and their state (success, failure, pending). The manifest procedure allows the source system to recover from certain errors such as failure of network, source/host system or in the transfer process. In the event of an error, the source system would know to attempt a redeliver for any file that had not received a successful "226" code from Openbridge.

System Status Codes

Your client will normally be sent a response code of "226" to indicate a successful file transfer. However, there are other status codes that may be present. See below:

  • A code 226 is sent if the entire file was successfully received and stored

  • A code 331 is sent due to a login error. The name is okay, but you are missing a password.

  • A code 425 is sent if no TCP connection was established. This might mean the server is not available or some other network error. Change from PASV to PORT mode, check your firewall settings

  • A code 426 is sent if the TCP connection was established but then broken by the client or by network failure. Retry later.

  • A request with code 451, 452, or 552 if the server had trouble saving the file to disk. Retry later.

  • A 553 code means the requested action not taken because the file name sent is not allowed. Change the file name or delete spaces/special characters in the file name.

If you are stuck with an error condition, reach out to Openbridge Support (support@openbridge.com) for help.

File Integrity

Openbridge uses checksum (or hashes) validation to ensure the integrity of the file transfer. Openbridge uses a 128-bit MD5 checksum, which is represented by a 32-character hexadecimal number. This only applies to files once they are in our possession. We suggest that source systems also calculate MD5 checksum in advance of file delivery to increase the confidence that the integrity of the file to be delivered is met. Depending the tools employed by a source system, the checksum process might be built-in.

Employing file integrity checks will help you cross check that the files delivered form the source system match what was delivered to Openbridge.

While the use of MD5 is the default, Openbridge can support other checksum commands when sending data to us:

  • XCRC (requests CRC32 digest/checksum)

  • XSHA/XSHA1 (requests SHA1 digest/checksum)

  • XSHA256 (requests SHA256 digest/checksum)

  • XSHA512 (requests SHA512 digest/checksum)

If you need assistance for any of these these, please contact Openbridge Support.

Connection Considerations

DNS-based Blackhole List (DNSBL) / Real-time Blackhole List (RBL)

Openbridge employs a DNSBL (commonly known as a 'Blocklist"). This is a database that is queried in realtime for the purpose of obtaining an opinion on the origin of incoming hosts. The role of a DNSBL is to assess whether a particular IP Address meets acceptance policies of inbound connections. DNSBL is often used by email, web and other network services for determining and rejecting addresses known to be sources of spam, phishing and other unwanted behavior.

More information on DNS blacklists can be found here:(http://en.wikipedia.org/wiki/DNSBL)

Anti-Virus, Malware and Trojans

Openbridge employs an anti-virus toolkit to scan for viruses, trojans, and other questionable items to prevent them from being uploaded to our system. The process is designed to detect in real-time threats present in any incoming files. This means any file sent to Openbridge will be scanned as it is streamed to us prior to allowing the file upload to be fully written to the filesystem.

Any files uploaded meeting the criteria as a threat will result the transfer being rejected.

Account Ban and Lockout

Openbridge employs a dynamic "ban" lists that prevents the banned user or host from logging in to the server. This will occur if our system detects 4 incorrect login attempts. The ban will last for approximately 30 minutes at which time you can attempt to login again. If you continue to have difficulties please contact support.

Idle Connection Time Limits

Openbridge sets the maximum number of seconds a connection between the our server and your client can exist after the client has successfully authenticated. This is typically 10 minutes or 600 seconds in length. If you are idle for longer than this time allotment the server will think you are finished and disconnect your client. If this occurs you will simply need to reconnect.

Did this answer your question?