训练模型最后遇到问题:math_functions.cu:375] Check failed: CURAND_STATUS_LAUNCH_FAILURE

用faster-rcnn训练自己的数据,最后出现错误如下:
I0917 18:33:26.278703  6169 layer_factory.hpp:77] Creating layer data
I0917 18:33:26.279088  6169 net.cpp:106] Creating Layer data
I0917 18:33:26.279096  6169 net.cpp:411] data -> data
I0917 18:33:26.279105  6169 net.cpp:411] data -> rois
I0917 18:33:26.279116  6169 net.cpp:411] data -> labels
I0917 18:33:26.279121  6169 net.cpp:411] data -> bbox_targets
I0917 18:33:26.279125  6169 net.cpp:411] data -> bbox_inside_weights
I0917 18:33:26.279129  6169 net.cpp:411] data -> bbox_outside_weights
RoiDataLayer: name_to_top: {'bbox_inside_weights': 4, 'labels': 2, 'rois': 1, 'bbox_targets': 3, 'bbox_outside_weights': 5, 'data': 0}
I0917 18:33:26.300452  6169 net.cpp:150] Setting up data
I0917 18:33:26.300473  6169 net.cpp:157] Top shape: 2 3 600 1000 (3600000)
I0917 18:33:26.300477  6169 net.cpp:157] Top shape: 1 5 (5)
I0917 18:33:26.300479  6169 net.cpp:157] Top shape: 1 (1)
I0917 18:33:26.300482  6169 net.cpp:157] Top shape: 1 8 (8)
I0917 18:33:26.300485  6169 net.cpp:157] Top shape: 1 8 (8)
I0917 18:33:26.300488  6169 net.cpp:157] Top shape: 1 8 (8)
I0917 18:33:26.300490  6169 net.cpp:165] Memory required for data: 14400120
I0917 18:33:26.300494  6169 layer_factory.hpp:77] Creating layer conv1
I0917 18:33:26.300511  6169 net.cpp:106] Creating Layer conv1
I0917 18:33:26.300516  6169 net.cpp:454] conv1 <- data
I0917 18:33:26.300521  6169 net.cpp:411] conv1 -> conv1
I0917 18:33:26.301926  6169 net.cpp:150] Setting up conv1
I0917 18:33:26.301939  6169 net.cpp:157] Top shape: 2 96 300 500 (28800000)
I0917 18:33:26.301941  6169 net.cpp:165] Memory required for data: 129600120
I0917 18:33:26.301951  6169 layer_factory.hpp:77] Creating layer relu1
I0917 18:33:26.301957  6169 net.cpp:106] Creating Layer relu1
I0917 18:33:26.301960  6169 net.cpp:454] relu1 <- conv1
I0917 18:33:26.301964  6169 net.cpp:397] relu1 -> conv1 (in-place)
I0917 18:33:26.301975  6169 net.cpp:150] Setting up relu1
I0917 18:33:26.301977  6169 net.cpp:157] Top shape: 2 96 300 500 (28800000)
I0917 18:33:26.301980  6169 net.cpp:165] Memory required for data: 244800120
I0917 18:33:26.301982  6169 layer_factory.hpp:77] Creating layer norm1
I0917 18:33:26.301988  6169 net.cpp:106] Creating Layer norm1
I0917 18:33:26.301990  6169 net.cpp:454] norm1 <- conv1
I0917 18:33:26.301995  6169 net.cpp:411] norm1 -> norm1
I0917 18:33:26.302069  6169 net.cpp:150] Setting up norm1
I0917 18:33:26.302074  6169 net.cpp:157] Top shape: 2 96 300 500 (28800000)
I0917 18:33:26.302078  6169 net.cpp:165] Memory required for data: 360000120
I0917 18:33:26.302079  6169 layer_factory.hpp:77] Creating layer pool1
I0917 18:33:26.302084  6169 net.cpp:106] Creating Layer pool1
I0917 18:33:26.302086  6169 net.cpp:454] pool1 <- norm1
I0917 18:33:26.302090  6169 net.cpp:411] pool1 -> pool1
I0917 18:33:26.302111  6169 net.cpp:150] Setting up pool1
I0917 18:33:26.302115  6169 net.cpp:157] Top shape: 2 96 151 251 (7276992)
I0917 18:33:26.302119  6169 net.cpp:165] Memory required for data: 389108088
I0917 18:33:26.302120  6169 layer_factory.hpp:77] Creating layer conv2
I0917 18:33:26.302125  6169 net.cpp:106] Creating Layer conv2
I0917 18:33:26.302129  6169 net.cpp:454] conv2 <- pool1
I0917 18:33:26.302132  6169 net.cpp:411] conv2 -> conv2
I0917 18:33:26.303267  6169 net.cpp:150] Setting up conv2
I0917 18:33:26.303279  6169 net.cpp:157] Top shape: 2 256 76 126 (4902912)
I0917 18:33:26.303282  6169 net.cpp:165] Memory required for data: 408719736
I0917 18:33:26.303289  6169 layer_factory.hpp:77] Creating layer relu2
I0917 18:33:26.303294  6169 net.cpp:106] Creating Layer relu2
I0917 18:33:26.303297  6169 net.cpp:454] relu2 <- conv2
I0917 18:33:26.303302  6169 net.cpp:397] relu2 -> conv2 (in-place)
I0917 18:33:26.303308  6169 net.cpp:150] Setting up relu2
I0917 18:33:26.303311  6169 net.cpp:157] Top shape: 2 256 76 126 (4902912)
I0917 18:33:26.303313  6169 net.cpp:165] Memory required for data: 428331384
I0917 18:33:26.303316  6169 layer_factory.hpp:77] Creating layer norm2
I0917 18:33:26.303320  6169 net.cpp:106] Creating Layer norm2
I0917 18:33:26.303323  6169 net.cpp:454] norm2 <- conv2
I0917 18:33:26.303325  6169 net.cpp:411] norm2 -> norm2
I0917 18:33:26.303383  6169 net.cpp:150] Setting up norm2
I0917 18:33:26.303390  6169 net.cpp:157] Top shape: 2 256 76 126 (4902912)
I0917 18:33:26.303391  6169 net.cpp:165] Memory required for data: 447943032
I0917 18:33:26.303393  6169 layer_factory.hpp:77] Creating layer pool2
I0917 18:33:26.303398  6169 net.cpp:106] Creating Layer pool2
I0917 18:33:26.303401  6169 net.cpp:454] pool2 <- norm2
I0917 18:33:26.303406  6169 net.cpp:411] pool2 -> pool2
I0917 18:33:26.303424  6169 net.cpp:150] Setting up pool2
I0917 18:33:26.303428  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.303431  6169 net.cpp:165] Memory required for data: 453054840
I0917 18:33:26.303433  6169 layer_factory.hpp:77] Creating layer conv3
I0917 18:33:26.303438  6169 net.cpp:106] Creating Layer conv3
I0917 18:33:26.303442  6169 net.cpp:454] conv3 <- pool2
I0917 18:33:26.303445  6169 net.cpp:411] conv3 -> conv3
I0917 18:33:26.306082  6169 net.cpp:150] Setting up conv3
I0917 18:33:26.306094  6169 net.cpp:157] Top shape: 2 384 39 64 (1916928)
I0917 18:33:26.306098  6169 net.cpp:165] Memory required for data: 460722552
I0917 18:33:26.306105  6169 layer_factory.hpp:77] Creating layer relu3
I0917 18:33:26.306110  6169 net.cpp:106] Creating Layer relu3
I0917 18:33:26.306113  6169 net.cpp:454] relu3 <- conv3
I0917 18:33:26.306118  6169 net.cpp:397] relu3 -> conv3 (in-place)
I0917 18:33:26.306124  6169 net.cpp:150] Setting up relu3
I0917 18:33:26.306128  6169 net.cpp:157] Top shape: 2 384 39 64 (1916928)
I0917 18:33:26.306129  6169 net.cpp:165] Memory required for data: 468390264
I0917 18:33:26.306133  6169 layer_factory.hpp:77] Creating layer conv4
I0917 18:33:26.306138  6169 net.cpp:106] Creating Layer conv4
I0917 18:33:26.306139  6169 net.cpp:454] conv4 <- conv3
I0917 18:33:26.306143  6169 net.cpp:411] conv4 -> conv4
I0917 18:33:26.308182  6169 net.cpp:150] Setting up conv4
I0917 18:33:26.308195  6169 net.cpp:157] Top shape: 2 384 39 64 (1916928)
I0917 18:33:26.308198  6169 net.cpp:165] Memory required for data: 476057976
I0917 18:33:26.308203  6169 layer_factory.hpp:77] Creating layer relu4
I0917 18:33:26.308207  6169 net.cpp:106] Creating Layer relu4
I0917 18:33:26.308210  6169 net.cpp:454] relu4 <- conv4
I0917 18:33:26.308215  6169 net.cpp:397] relu4 -> conv4 (in-place)
I0917 18:33:26.308220  6169 net.cpp:150] Setting up relu4
I0917 18:33:26.308223  6169 net.cpp:157] Top shape: 2 384 39 64 (1916928)
I0917 18:33:26.308225  6169 net.cpp:165] Memory required for data: 483725688
I0917 18:33:26.308228  6169 layer_factory.hpp:77] Creating layer conv5
I0917 18:33:26.308233  6169 net.cpp:106] Creating Layer conv5
I0917 18:33:26.308236  6169 net.cpp:454] conv5 <- conv4
I0917 18:33:26.308240  6169 net.cpp:411] conv5 -> conv5
I0917 18:33:26.309712  6169 net.cpp:150] Setting up conv5
I0917 18:33:26.309723  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.309726  6169 net.cpp:165] Memory required for data: 488837496
I0917 18:33:26.309733  6169 layer_factory.hpp:77] Creating layer relu5
I0917 18:33:26.309739  6169 net.cpp:106] Creating Layer relu5
I0917 18:33:26.309742  6169 net.cpp:454] relu5 <- conv5
I0917 18:33:26.309746  6169 net.cpp:397] relu5 -> conv5 (in-place)
I0917 18:33:26.309751  6169 net.cpp:150] Setting up relu5
I0917 18:33:26.309754  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.309756  6169 net.cpp:165] Memory required for data: 493949304
I0917 18:33:26.309758  6169 layer_factory.hpp:77] Creating layer conv5_relu5_0_split
I0917 18:33:26.309763  6169 net.cpp:106] Creating Layer conv5_relu5_0_split
I0917 18:33:26.309765  6169 net.cpp:454] conv5_relu5_0_split <- conv5
I0917 18:33:26.309769  6169 net.cpp:411] conv5_relu5_0_split -> conv5_relu5_0_split_0
I0917 18:33:26.309773  6169 net.cpp:411] conv5_relu5_0_split -> conv5_relu5_0_split_1
I0917 18:33:26.309798  6169 net.cpp:150] Setting up conv5_relu5_0_split
I0917 18:33:26.309801  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.309804  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.309806  6169 net.cpp:165] Memory required for data: 504172920
I0917 18:33:26.309809  6169 layer_factory.hpp:77] Creating layer roi_pool_conv5
I0917 18:33:26.309815  6169 net.cpp:106] Creating Layer roi_pool_conv5
I0917 18:33:26.309818  6169 net.cpp:454] roi_pool_conv5 <- conv5_relu5_0_split_0
I0917 18:33:26.309823  6169 net.cpp:454] roi_pool_conv5 <- rois
I0917 18:33:26.309825  6169 net.cpp:411] roi_pool_conv5 -> roi_pool_conv5
I0917 18:33:26.309830  6169 roi_pooling_layer.cpp:30] Spatial scale: 0.0625
I0917 18:33:26.309859  6169 net.cpp:150] Setting up roi_pool_conv5
I0917 18:33:26.309862  6169 net.cpp:157] Top shape: 1 256 6 6 (9216)
I0917 18:33:26.309864  6169 net.cpp:165] Memory required for data: 504209784
I0917 18:33:26.309867  6169 layer_factory.hpp:77] Creating layer fc6
I0917 18:33:26.309872  6169 net.cpp:106] Creating Layer fc6
I0917 18:33:26.309875  6169 net.cpp:454] fc6 <- roi_pool_conv5
I0917 18:33:26.309880  6169 net.cpp:411] fc6 -> fc6
I0917 18:33:26.371209  6169 net.cpp:150] Setting up fc6
I0917 18:33:26.371243  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.371245  6169 net.cpp:165] Memory required for data: 504226168
I0917 18:33:26.371255  6169 layer_factory.hpp:77] Creating layer relu6
I0917 18:33:26.371263  6169 net.cpp:106] Creating Layer relu6
I0917 18:33:26.371268  6169 net.cpp:454] relu6 <- fc6
I0917 18:33:26.371274  6169 net.cpp:397] relu6 -> fc6 (in-place)
I0917 18:33:26.371284  6169 net.cpp:150] Setting up relu6
I0917 18:33:26.371287  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.371289  6169 net.cpp:165] Memory required for data: 504242552
I0917 18:33:26.371292  6169 layer_factory.hpp:77] Creating layer drop6
I0917 18:33:26.371297  6169 net.cpp:106] Creating Layer drop6
I0917 18:33:26.371300  6169 net.cpp:454] drop6 <- fc6
I0917 18:33:26.371304  6169 net.cpp:397] drop6 -> fc6 (in-place)
I0917 18:33:26.371320  6169 net.cpp:150] Setting up drop6
I0917 18:33:26.371325  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.371326  6169 net.cpp:165] Memory required for data: 504258936
I0917 18:33:26.371328  6169 layer_factory.hpp:77] Creating layer fc7
I0917 18:33:26.371335  6169 net.cpp:106] Creating Layer fc7
I0917 18:33:26.371337  6169 net.cpp:454] fc7 <- fc6
I0917 18:33:26.371341  6169 net.cpp:411] fc7 -> fc7
I0917 18:33:26.398211  6169 net.cpp:150] Setting up fc7
I0917 18:33:26.398244  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.398248  6169 net.cpp:165] Memory required for data: 504275320
I0917 18:33:26.398257  6169 layer_factory.hpp:77] Creating layer relu7
I0917 18:33:26.398267  6169 net.cpp:106] Creating Layer relu7
I0917 18:33:26.398272  6169 net.cpp:454] relu7 <- fc7
I0917 18:33:26.398278  6169 net.cpp:397] relu7 -> fc7 (in-place)
I0917 18:33:26.398288  6169 net.cpp:150] Setting up relu7
I0917 18:33:26.398290  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.398293  6169 net.cpp:165] Memory required for data: 504291704
I0917 18:33:26.398295  6169 layer_factory.hpp:77] Creating layer drop7
I0917 18:33:26.398300  6169 net.cpp:106] Creating Layer drop7
I0917 18:33:26.398303  6169 net.cpp:454] drop7 <- fc7
I0917 18:33:26.398306  6169 net.cpp:397] drop7 -> fc7 (in-place)
I0917 18:33:26.398324  6169 net.cpp:150] Setting up drop7
I0917 18:33:26.398327  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.398329  6169 net.cpp:165] Memory required for data: 504308088
I0917 18:33:26.398332  6169 layer_factory.hpp:77] Creating layer fc7_drop7_0_split
I0917 18:33:26.398337  6169 net.cpp:106] Creating Layer fc7_drop7_0_split
I0917 18:33:26.398339  6169 net.cpp:454] fc7_drop7_0_split <- fc7
I0917 18:33:26.398344  6169 net.cpp:411] fc7_drop7_0_split -> fc7_drop7_0_split_0
I0917 18:33:26.398349  6169 net.cpp:411] fc7_drop7_0_split -> fc7_drop7_0_split_1
I0917 18:33:26.398372  6169 net.cpp:150] Setting up fc7_drop7_0_split
I0917 18:33:26.398376  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.398380  6169 net.cpp:157] Top shape: 1 4096 (4096)
I0917 18:33:26.398381  6169 net.cpp:165] Memory required for data: 504340856
I0917 18:33:26.398385  6169 layer_factory.hpp:77] Creating layer cls_score
I0917 18:33:26.398391  6169 net.cpp:106] Creating Layer cls_score
I0917 18:33:26.398393  6169 net.cpp:454] cls_score <- fc7_drop7_0_split_0
I0917 18:33:26.398398  6169 net.cpp:411] cls_score -> cls_score
I0917 18:33:26.398676  6169 net.cpp:150] Setting up cls_score
I0917 18:33:26.398684  6169 net.cpp:157] Top shape: 1 2 (2)
I0917 18:33:26.398686  6169 net.cpp:165] Memory required for data: 504340864
I0917 18:33:26.398691  6169 layer_factory.hpp:77] Creating layer bbox_pred
I0917 18:33:26.398696  6169 net.cpp:106] Creating Layer bbox_pred
I0917 18:33:26.398699  6169 net.cpp:454] bbox_pred <- fc7_drop7_0_split_1
I0917 18:33:26.398704  6169 net.cpp:411] bbox_pred -> bbox_pred
I0917 18:33:26.401015  6169 net.cpp:150] Setting up bbox_pred
I0917 18:33:26.401027  6169 net.cpp:157] Top shape: 1 8 (8)
I0917 18:33:26.401031  6169 net.cpp:165] Memory required for data: 504340896
I0917 18:33:26.401041  6169 layer_factory.hpp:77] Creating layer loss_cls
I0917 18:33:26.401051  6169 net.cpp:106] Creating Layer loss_cls
I0917 18:33:26.401056  6169 net.cpp:454] loss_cls <- cls_score
I0917 18:33:26.401059  6169 net.cpp:454] loss_cls <- labels
I0917 18:33:26.401063  6169 net.cpp:411] loss_cls -> cls_loss
I0917 18:33:26.401073  6169 layer_factory.hpp:77] Creating layer loss_cls
I0917 18:33:26.401134  6169 net.cpp:150] Setting up loss_cls
I0917 18:33:26.401139  6169 net.cpp:157] Top shape: (1)
I0917 18:33:26.401141  6169 net.cpp:160]     with loss weight 1
I0917 18:33:26.401150  6169 net.cpp:165] Memory required for data: 504340900
I0917 18:33:26.401152  6169 layer_factory.hpp:77] Creating layer loss_bbox
I0917 18:33:26.401160  6169 net.cpp:106] Creating Layer loss_bbox
I0917 18:33:26.401162  6169 net.cpp:454] loss_bbox <- bbox_pred
I0917 18:33:26.401166  6169 net.cpp:454] loss_bbox <- bbox_targets
I0917 18:33:26.401170  6169 net.cpp:454] loss_bbox <- bbox_inside_weights
I0917 18:33:26.401172  6169 net.cpp:454] loss_bbox <- bbox_outside_weights
I0917 18:33:26.401176  6169 net.cpp:411] loss_bbox -> bbox_loss
I0917 18:33:26.401228  6169 net.cpp:150] Setting up loss_bbox
I0917 18:33:26.401235  6169 net.cpp:157] Top shape: (1)
I0917 18:33:26.401237  6169 net.cpp:160]     with loss weight 1
I0917 18:33:26.401242  6169 net.cpp:165] Memory required for data: 504340904
I0917 18:33:26.401243  6169 layer_factory.hpp:77] Creating layer rpn_conv1
I0917 18:33:26.401252  6169 net.cpp:106] Creating Layer rpn_conv1
I0917 18:33:26.401255  6169 net.cpp:454] rpn_conv1 <- conv5_relu5_0_split_1
I0917 18:33:26.401262  6169 net.cpp:411] rpn_conv1 -> rpn_conv1
I0917 18:33:26.416738  6169 net.cpp:150] Setting up rpn_conv1
I0917 18:33:26.416750  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.416754  6169 net.cpp:165] Memory required for data: 509452712
I0917 18:33:26.416759  6169 layer_factory.hpp:77] Creating layer rpn_relu1
I0917 18:33:26.416764  6169 net.cpp:106] Creating Layer rpn_relu1
I0917 18:33:26.416766  6169 net.cpp:454] rpn_relu1 <- rpn_conv1
I0917 18:33:26.416771  6169 net.cpp:397] rpn_relu1 -> rpn_conv1 (in-place)
I0917 18:33:26.416776  6169 net.cpp:150] Setting up rpn_relu1
I0917 18:33:26.416779  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.416782  6169 net.cpp:165] Memory required for data: 514564520
I0917 18:33:26.416785  6169 layer_factory.hpp:77] Creating layer rpn_conv1_rpn_relu1_0_split
I0917 18:33:26.416787  6169 net.cpp:106] Creating Layer rpn_conv1_rpn_relu1_0_split
I0917 18:33:26.416790  6169 net.cpp:454] rpn_conv1_rpn_relu1_0_split <- rpn_conv1
I0917 18:33:26.416795  6169 net.cpp:411] rpn_conv1_rpn_relu1_0_split -> rpn_conv1_rpn_relu1_0_split_0
I0917 18:33:26.416798  6169 net.cpp:411] rpn_conv1_rpn_relu1_0_split -> rpn_conv1_rpn_relu1_0_split_1
I0917 18:33:26.416822  6169 net.cpp:150] Setting up rpn_conv1_rpn_relu1_0_split
I0917 18:33:26.416826  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.416829  6169 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I0917 18:33:26.416831  6169 net.cpp:165] Memory required for data: 524788136
I0917 18:33:26.416834  6169 layer_factory.hpp:77] Creating layer rpn_cls_score
I0917 18:33:26.416841  6169 net.cpp:106] Creating Layer rpn_cls_score
I0917 18:33:26.416844  6169 net.cpp:454] rpn_cls_score <- rpn_conv1_rpn_relu1_0_split_0
I0917 18:33:26.416849  6169 net.cpp:411] rpn_cls_score -> rpn_cls_score
I0917 18:33:26.417110  6169 net.cpp:150] Setting up rpn_cls_score
I0917 18:33:26.417119  6169 net.cpp:157] Top shape: 2 18 39 64 (89856)
I0917 18:33:26.417121  6169 net.cpp:165] Memory required for data: 525147560
I0917 18:33:26.417125  6169 layer_factory.hpp:77] Creating layer rpn_bbox_pred
I0917 18:33:26.417138  6169 net.cpp:106] Creating Layer rpn_bbox_pred
I0917 18:33:26.417141  6169 net.cpp:454] rpn_bbox_pred <- rpn_conv1_rpn_relu1_0_split_1
I0917 18:33:26.417146  6169 net.cpp:411] rpn_bbox_pred -> rpn_bbox_pred
I0917 18:33:26.417526  6169 net.cpp:150] Setting up rpn_bbox_pred
I0917 18:33:26.417534  6169 net.cpp:157] Top shape: 2 36 39 64 (179712)
I0917 18:33:26.417536  6169 net.cpp:165] Memory required for data: 525866408
I0917 18:33:26.417541  6169 layer_factory.hpp:77] Creating layer silence_rpn_cls_score
I0917 18:33:26.417546  6169 net.cpp:106] Creating Layer silence_rpn_cls_score
I0917 18:33:26.417548  6169 net.cpp:454] silence_rpn_cls_score <- rpn_cls_score
I0917 18:33:26.417552  6169 net.cpp:150] Setting up silence_rpn_cls_score
I0917 18:33:26.417554  6169 net.cpp:165] Memory required for data: 525866408
I0917 18:33:26.417557  6169 layer_factory.hpp:77] Creating layer silence_rpn_bbox_pred
I0917 18:33:26.417559  6169 net.cpp:106] Creating Layer silence_rpn_bbox_pred
I0917 18:33:26.417562  6169 net.cpp:454] silence_rpn_bbox_pred <- rpn_bbox_pred
I0917 18:33:26.417564  6169 net.cpp:150] Setting up silence_rpn_bbox_pred
I0917 18:33:26.417567  6169 net.cpp:165] Memory required for data: 525866408
I0917 18:33:26.417569  6169 net.cpp:228] silence_rpn_bbox_pred does not need backward computation.
I0917 18:33:26.417572  6169 net.cpp:228] silence_rpn_cls_score does not need backward computation.
I0917 18:33:26.417574  6169 net.cpp:228] rpn_bbox_pred does not need backward computation.
I0917 18:33:26.417577  6169 net.cpp:228] rpn_cls_score does not need backward computation.
I0917 18:33:26.417579  6169 net.cpp:228] rpn_conv1_rpn_relu1_0_split does not need backward computation.
I0917 18:33:26.417582  6169 net.cpp:228] rpn_relu1 does not need backward computation.
I0917 18:33:26.417584  6169 net.cpp:228] rpn_conv1 does not need backward computation.
I0917 18:33:26.417587  6169 net.cpp:226] loss_bbox needs backward computation.
I0917 18:33:26.417589  6169 net.cpp:226] loss_cls needs backward computation.
I0917 18:33:26.417593  6169 net.cpp:226] bbox_pred needs backward computation.
I0917 18:33:26.417595  6169 net.cpp:226] cls_score needs backward computation.
I0917 18:33:26.417598  6169 net.cpp:226] fc7_drop7_0_split needs backward computation.
I0917 18:33:26.417600  6169 net.cpp:226] drop7 needs backward computation.
I0917 18:33:26.417603  6169 net.cpp:226] relu7 needs backward computation.
I0917 18:33:26.417604  6169 net.cpp:226] fc7 needs backward computation.
I0917 18:33:26.417608  6169 net.cpp:226] drop6 needs backward computation.
I0917 18:33:26.417609  6169 net.cpp:226] relu6 needs backward computation.
I0917 18:33:26.417611  6169 net.cpp:226] fc6 needs backward computation.
I0917 18:33:26.417614  6169 net.cpp:226] roi_pool_conv5 needs backward computation.
I0917 18:33:26.417618  6169 net.cpp:226] conv5_relu5_0_split needs backward computation.
I0917 18:33:26.417620  6169 net.cpp:226] relu5 needs backward computation.
I0917 18:33:26.417623  6169 net.cpp:226] conv5 needs backward computation.
I0917 18:33:26.417625  6169 net.cpp:226] relu4 needs backward computation.
I0917 18:33:26.417628  6169 net.cpp:226] conv4 needs backward computation.
I0917 18:33:26.417630  6169 net.cpp:226] relu3 needs backward computation.
I0917 18:33:26.417632  6169 net.cpp:226] conv3 needs backward computation.
I0917 18:33:26.417634  6169 net.cpp:226] pool2 needs backward computation.
I0917 18:33:26.417637  6169 net.cpp:226] norm2 needs backward computation.
I0917 18:33:26.417639  6169 net.cpp:226] relu2 needs backward computation.
I0917 18:33:26.417642  6169 net.cpp:226] conv2 needs backward computation.
I0917 18:33:26.417644  6169 net.cpp:226] pool1 needs backward computation.
I0917 18:33:26.417647  6169 net.cpp:226] norm1 needs backward computation.
I0917 18:33:26.417649  6169 net.cpp:226] relu1 needs backward computation.
I0917 18:33:26.417651  6169 net.cpp:226] conv1 needs backward computation.
I0917 18:33:26.417655  6169 net.cpp:228] data does not need backward computation.
I0917 18:33:26.417657  6169 net.cpp:270] This network produces output bbox_loss
I0917 18:33:26.417660  6169 net.cpp:270] This network produces output cls_loss
I0917 18:33:26.417680  6169 net.cpp:283] Network initialization done.
I0917 18:33:26.417758  6169 solver.cpp:60] Solver scaffolding done.
Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
I0917 18:33:26.722081  6169 net.cpp:816] Ignoring source layer pool5_spm6
I0917 18:33:26.722110  6169 net.cpp:816] Ignoring source layer pool5_spm6_flatten
I0917 18:33:26.756674  6169 net.cpp:816] Ignoring source layer fc8
I0917 18:33:26.756700  6169 net.cpp:816] Ignoring source layer prob
Solving...
F0917 18:33:28.226620  6169 math_functions.cu:375] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE

看到nvidia官方解释说:是因为执行随机数生成函数时候,内核启动失败,不知道是什么 导致的。求解!!!
已邀请:

Paul_H

赞同来自:

我也碰到了一样的问题,请问楼主解决了吗?

aleexyang - 深度学习在读研究生

赞同来自:

题主,请问你仅仅是在训练F-rcnn出现了这个问题么?以前跑别的模型的时候有没有出过同样的的问题?

HmyBmny - 深度学习 小学生

赞同来自:

应该是 GPU 内存不够用,把 batch_size 改小一点看下,实在不行就使用 CPU 吧。

aleexyang - 深度学习在读研究生

赞同来自:

题主,问题解决了,的确是和版本有关的。更新了CUDA8.0之后就可以正常的运行了。我个人猜测可能是7.5版本对于10系列的显卡的支持还不是很完善,可能在有些时候会出问题。

cugxyy

赞同来自:

楼主您好,请问您现在针对faster rcnn使用的是什么显卡?方便留邮箱我吗?谢谢 

要回复问题请先登录注册