tensorflow: Running tensorflow on GPU is far slower than on CPU

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 1809 & Windows Server 2016
• TensorFlow installed from (source or binary): pip install tensorflow-gpu==2.0.0-beta1, as well as tensorflow-gpu, compared to tensorflow & tensorflow==2.0.0-beta1
• Python version: 3.6
• Bazel version (if compiling from source): -
• GCC/Compiler version (if compiling from source): -
• CUDA/cuDNN version: Cuda 10 & cuDNN 7.6.2 for cuda 10.0

Current behavior:

Im getting a 50%+ performance loss with GPU! In the below example, the CPU version is even training way faster on a bigger model with slightly bigger epochs.

Im training on 2 different systems: My server, without a GPU:

Intel Core i5 6500T (4x@2.5 ghz) Notebook processor Network attached storage for training data and output 16 Gb DDR 4 Ram My Desktop PC:

Intel core i7 3770k (4x3,5-4 ghz) Nvidia GTX 970 @4GB 32gb ddr3 ram Training Data on local SSD, output to NAS

Now interestingly 3000 epochs, 100000 records each, takes roughly 3h on the server using TF 1.14 The same on my Desktop with GPU takes 8h with TF 2.0 It sits with full Video Ram but at 3% graphical processor use. The CPU is sometimes at 30% use with tensorflow GPU but 100% at any time with any CPU build.
The harddisk is utilized with a whopping 0%.

Expected behavior:

Tensorflow-GPU trains faster than Tensorflow CPU

Code to reproduce the issue:

My model is a fairly simple keras sequential lstm:

for learningrate in learningrates:
	for layerdensity in layerdensitys:
		for layer in amount_of_layers:
			################################
			# generate model               #
			################################
			modelname = f"{layer}-layer_{layerdensity}-nodes_selu-adam_{learningrate}-learningrate_{records_per_epoch}-epochsize_{appendix}"
			model = keras.Sequential()
			model.add(Dense(layerdensity, activation=tf.nn.selu, input_dim=15))
			for i in range(layer-1):
				model.add(Dense(layerdensity, activation=tf.nn.selu))
			model.add(Dense(9,activation=tf.nn.softmax, name = "Output"))
			# Compile
			optimizer = tf.keras.optimizers.Adam(lr=learningrate)
			model.compile(
				optimizer=optimizer,
				loss='sparse_categorical_crossentropy',
				metrics=['accuracy'])
			model.summary()
			tensorboard = TensorBoard(log_dir="\\\\drg-fs01\\BigData\\Projects\\Notebooks\\PokerBot\\log\\" + modelname,
				histogram_freq = 100, write_graph = False)
			#cp_callback = tf.keras.callbacks.ModelCheckpoint("\\\\drg-fs01\\BigData\\Projects\\Notebooks\\PokerBot\\checkpoints\\" + modelname, verbose=0)
			################################
			# train model                  #
			################################
			model.fit(trainSet, 
				epochs = epochs, 
				steps_per_epoch = trainSteps, 
				shuffle = True, 
				validation_data = testSet, 
				validation_steps = testSteps, 
				validation_freq = int(epochs/maxTestEpochs),
				verbose = verbose, 
				callbacks = [tensorboard])#,cp_callback])
			model.save(basePath+'saved_models/' + modelname + '.h5')

I am having lots of test data which does not fit in memory so I use interleaved datasets:

# dataset modeler
def modelDataset(sourcepath, badgesize, repeat = False, repetitions = 10):
    #get all files
    list = os.listdir(sourcepath)
    pathfiles = [sourcepath+x for x in list]
    
    #get metrics
    rows_per_file = count_lines(sourcepath+"0.csv")
    number_of_files = len(list)
    total_rows = (rows_per_file * number_of_files)
    print(f"records: {total_rows}")
    # get number of steps per Epoch
    steps_per_epoch = int(rows_per_file / badgesize) # 2000 badges per epoch
    epochs = number_of_files
    if badgesize == 1:
        epochs = 1 
    print(f"number epochs: {epochs}")
    # model interleaved dataset
    dataset = (tf.data.Dataset.from_tensor_slices(pathfiles).interleave(lambda x:
        tf.data.TextLineDataset(x).map(parse_csv, num_parallel_calls=4),
        cycle_length=4, block_length=16))
    dataset.columns = CSV_COLUMNS
    
    if badgesize != 1:
        dataset = dataset.shuffle(buffer_size=badgesize)
    if repeat:
        dataset = dataset.repeat(repetitions)
        epochs = epochs * repetitions
    dataset = dataset.batch(badgesize)
    dataset = dataset.prefetch(2)  # prefetch one batch
    return dataset, steps_per_epoch, epochs, badgesize