hoverfly: Slow import

I have a large export file (~500Mb). Starting hoverfly with “hoverfly -import simulation.json” takes around 2 hours for me:( I did a profiling and that what I see in pprof top:

      flat  flat%   sum%        cum   cum%
 14.73mins 11.62% 11.62%  91.20mins 71.95%  reflect.deepValueEqual
  7.59mins  5.99% 17.61%   8.91mins  7.03%  runtime.ifaceeq
  7.08mins  5.59% 23.20%  26.54mins 20.94%  runtime.mallocgc
  5.57mins  4.39% 27.59%   5.57mins  4.39%  runtime.memmove
  4.92mins  3.88% 31.47%   8.48mins  6.69%  runtime.strequal
  4.84mins  3.82% 35.29%  12.81mins 10.11%  runtime.mapassign
  4.49mins  3.54% 38.84%   5.84mins  4.61%  runtime.heapBitsSetType
  4.32mins  3.40% 42.24%   9.85mins  7.77%  runtime.scanobject
  4.06mins  3.20% 45.44%   4.06mins  3.20%  reflect.Value.Type
  3.21mins  2.53% 47.98%   3.21mins  2.53%  runtime.memclrNoHeapPointers

This lead me to the function AddPair in core/models/simulation.go, that is doing DeepEqual check for each pair adding to the simulation. I think its need to be optimized! When I add a new function “AddPairWithoutDuplicateCheck” and use it in core/import.go instead “AddPair” then import of the same file took few seconds.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (9 by maintainers)

Commits related to this issue

#848 Add benchmark test for PUT simulation API — committed to SpectoLabs/hoverfly by tommysitu 5 years ago
#848 Add hoverfly flag to set SkipImportCheck config — committed to SpectoLabs/hoverfly by tommysitu 5 years ago
#848 Add hoverctl flag to disable import check — committed to SpectoLabs/hoverfly by tommysitu 5 years ago
#848 Add benchmark test for PUT simulation API — committed to SpectoLabs/hoverfly by tommysitu 5 years ago
#848 Add hoverfly flag to set SkipImportCheck config — committed to SpectoLabs/hoverfly by tommysitu 5 years ago
#848 Add hoverctl flag to disable import check — committed to SpectoLabs/hoverfly by tommysitu 5 years ago

Most upvoted comments

Variant with flag fine with me. But was you benchmarking import? I can if you want provide a simulation file for you (replacing first sencitive data inside), that take hours to import, and seconds if I’m removing DeepEqual check.

here is quick and dirty fix that I’m using localy:

diff --git a/core/models/simulation.go b/core/models/simulation.go
index 4c8f1c5..3d9e906 100644
--- a/core/models/simulation.go
+++ b/core/models/simulation.go
@@ -42,6 +42,12 @@
 	return !duplicate
 }
 
+func (this *Simulation) AddPairWithoutDuplicateCheck(pair *RequestMatcherResponsePair) {
+	this.RWMutex.Lock()
+	this.matchingPairs = append(this.matchingPairs, *pair)
+	this.RWMutex.Unlock()
+}
+
 func (this *Simulation) AddPairInSequence(pair *RequestMatcherResponsePair, state *state.State) {
 	var duplicate bool

diff --git a/core/import.go b/core/import.go
index 95fcacf..76c1b3b 100644
--- a/core/import.go
+++ b/core/import.go
@@ -143,15 +143,11 @@
 
 			pair := models.NewRequestMatcherResponsePairFromView(&pairView)
 
-			isAdded := hf.Simulation.AddPair(pair)
-			if isAdded {
-				for k, v := range pair.RequestMatcher.RequiresState {
-					initialStates[k] = v
-				}
-				success++
-			} else {
-				importResult.AddPairIgnoredWarning(i)
+			hf.Simulation.AddPairWithoutDuplicateCheck(pair)
+			for k, v := range pair.RequestMatcher.RequiresState {
+				initialStates[k] = v
 			}
+			success++
 
 			if pairView.RequestMatcher.DeprecatedQuery != nil && len(pairView.RequestMatcher.DeprecatedQuery) != 0 {
 				importResult.AddDeprecatedQueryWarning(i)

aosavitski on Jun 28, 2019

Does this check is needed at all during import? Is there a possibility that duplicate requests will be in import file?

aosavitski on Jun 18, 2019